[00:00:00] well, ask legoktm [00:00:15] I'm not sold on the idea of putting new stuff on IRC [00:00:22] Gloria: what should we do? [00:00:27] Dunno! [00:00:30] we have to maintain the existing streams, fine; but newer stuff should be published some other way [00:00:38] ori: For testing purposes maybe.. [00:00:38] use the sane naming from the beginning [00:00:47] But hacks => permenant... [00:00:59] it's probably better to not start doing it :D [00:01:08] the most popular choice is websockets [00:01:30] but there is remarkably little information online about how to harden and scale websocket streams [00:01:39] Popular in what sense? [00:01:59] everyone that i've talked to or chatted with is most excited about that [00:03:06] Because it's webscale? :-) [00:03:17] Which RFC covers this? [00:03:24] ori: why do people not like xmpp pub/sub? [00:04:02] Gloria: none, currently; the logging RFC is relevant to the APIs within core, but it does not have much to do without how the stream is ultimately published [00:04:41] legoktm: it's a heavyweight protocol and it's clearly moribund [00:04:55] and it's not easy to consume via javascript [00:05:28] oh, so people can build gadgets that use the feed [00:05:38] websockets make sense then [00:05:48] yeah, and other neat things like http://listen.hatnote.com/ [00:06:14] (which IIRC works by having a piece of middleware that consumes the IRC feed, cleans it up, and republishes it as a websocket stream) [00:06:45] ori: Well, not to be a dick, but... I think the summit agenda is based on actual RFCs. [00:07:18] That said, I haven't followed this issue too closely except to know that IRC can't easily be removed and that XMPP would remove some of its limitations. [00:07:36] Gloria: i'm not sure what you're saying. i'm not asking lego to wait before devising a solution for streaming the feed. [00:07:43] if he wants to take that on, he can go ahead. [00:07:57] one of the topics for discussion for the architecture summit is the logging infrastructure [00:08:00] I was responding to that. [00:08:10] yeah, maybe that's not relevant [00:08:20] Unless the topic for discussion is in an RFC on mediawiki.org, it'll have difficulty being formally discussed. [00:08:24] I think. [00:08:45] legoktm: why haven't you written an RfC yet? [00:08:51] That isn't to say that it won't be or shouldn't be discussed, just that the agenda is based on published RFCs. [00:08:57] i don't personally thing it's a good subject for RFCs [00:09:07] for an RFC, even [00:09:11] At some point, I think the summit might focus on "what is an RFC?" [00:09:24] Because that seems to be defined in a lot of different ways. [00:09:34] Some of which are narrow and I don't really approve of. [00:09:53] someone who knows what their doing (or is committed to learning) should survey the options, choose one, and implement it [00:10:12] What are the hard and soft requirements? [00:10:59] well, let me answer that by backtracking for a moment and giving some more detail about the relevant considerations [00:11:00] I guess the person who knows what they're doing would be able to answer that. [00:11:24] i know what i'm doing, but i can't commit to that project [00:11:47] anyways: the thing that XMPP has going for it is that it's easy to scale. there are public, single-server ejabberd instances with tens of thousands of concurrent clients [00:11:56] facebook used ejabberd to power facebook chat for a while [00:12:17] it's written in erlang so it's architected for scale out, you can just add severs [00:13:34] back when Krenair revised vvv's patch and there seemed to be some momentum i emailed the ops list about it (krenair on CC) and mark replied saying xmpp would be a good choice, so it has that going for it too [00:14:23] Mark B>? [00:14:26] B. [00:14:30] yeah [00:15:16] around that time i also looked for information about how to harden and scale websockets, and it seemed like node.js behind nginx was the most popular setup [00:15:47] i emailed http://pusher.com/ to ask them if they can share any details or offer advice [00:16:30] they mentioned that but they also indicated they ran into all sorts of problems that they had to fix, but wouldn't disclose the details. understandably, since scaling websockets is their business model. [00:17:21] I think I've got a Pusher t-shirt [00:17:45] i think I met some of their devs [00:17:57] they're nice, and it's a cool service [00:18:47] so there are two viable options, IMO: [00:19:30] publish the stream using XMPP pub/sub, and rely on third-parties to consume the stream and rebroadcast it using convenient protocols [00:19:35] And if it is them I met... [00:19:48] They did some stuff using our irc rc that weekend.. [00:20:00] yeah, i think they had a demo stream or something [00:20:02] PubSubHubBub [00:20:11] pubsubhubbub is so dead [00:20:20] i considered it too [00:20:50] the protocol envisioned a future of federated hubs, afaik the only ones that survive are google's and superfeedr's. [00:21:06] I just like typing/reading that word. [00:21:19] and google's says something like "(c) 2011 google inc" [00:21:28] like no one got around to turning off the server [00:21:41] Or it's stable. [00:21:54] And well-tested. [00:22:11] it is stable and well tested, but it's not better than xmpp [00:22:16] even xmpp has brighter prospects [00:22:33] Let's write our own. [00:22:42] Was that option 2? [00:22:47] i bet you $5 that google's instance is shut down by the end of 2014 [00:22:51] Or pay pusher.com to do it. [00:23:12] we could, or superfeedr, or any of the other services [00:23:12] I'm not sure it's in my interest to take a bet that I have no interest in or knowledge about. [00:23:24] well, i don't think we should pay anyone, but we could ask them [00:23:31] it'd be a nice feather in their cap, i imagine [00:23:43] and if they are simply republishing the xmpp feed, then it's not horribly problematic [00:24:04] http://pusher.com/pricing [00:24:06] * Gloria NP: "Heartache Tonight" by Eagles from "The Long Run" [00:24:06] because the data would also be available using wikimedia infrastructure, just not in the most convenient format [00:24:08] 100k messages a day [00:24:14] NO SSL THOUGH!? [00:24:20] Reedy: srsly? [00:24:38] Not on the free one [00:24:38] http://pusher.com/pricing [00:24:40] option 2 is stop moaning about it and get to work figuring out how to set up nginx with nodejs [00:24:47] Why are you looking at the basic options? [00:24:53] Clearly Wikimedia would be enterprise level. [00:24:57] Because it's free [00:25:03] Or a Big Boy [00:25:09] Wikimedia has plenty of money. [00:25:16] Not that I think it should be spent on this, per se. [00:25:24] But it can afford $200/month. [00:25:48] i would probably prefer that we try it ourselves [00:25:53] I think using xmpp pub/sub and letting people republish it into whatever format they want is fine [00:26:08] There are now whole teams of people being paid that provide far less benefit than a small investment of $2400/year. [00:26:20] Gloria: don't hijack [00:26:31] that's an irrelevant tangent [00:26:38] Mods!!!!! [00:26:45] heh [00:27:00] just to flesh out option 2 a little: [00:27:03] If you want focus, write an RFC. I still don't see what the reluctance is. [00:27:24] the RC feed does not contain private data [00:27:39] "My ideas might be available in a readable, archivable, coherent, and discussable format. Heavens." [00:27:40] you could stream it using redis pubsub, or kafka, or zeromq, or even udp to a labs instance [00:27:43] Gloria: Behave, please. [00:27:57] and toy around with nginx + node.js there [00:28:32] in fact i recommend that regardless of what protocol you choose [00:29:07] oh yeah, last thing about ejabberd: already packaged for debian and in apt [00:29:14] Just as every cop is a criminal, and all the sinners saints, as heads is tails, just call me Lucifer, [00:29:26] apt.wm.o? [00:29:37] not even, it's in debian main [00:29:56] apt.wm.o is not a full mirror, it just contains wmf-specific packages [00:30:08] Interesting. [00:30:27] http://ubuntu.wikimedia.org/ is ;) [00:31:01] That's a new one to me. [00:31:30] i think my survey was reasonably thorough, and the requirements are pretty clear, so i don't see the point of having an RFC or discussing it at length. what is missing is initiative and follow-through [00:31:45] encumbering it with more opinions will just make it less likely to happen [00:31:55] If you say so. [00:32:02] I don't think there's much evidence to support that conclusion. [00:32:24] up to you, i'm just not motivated to write an up RFC myself [00:32:31] I'm writing something up [00:32:32] up an [00:32:42] Proper prior planning prevents piss-poor performance. [00:32:46] These colors don't run. [00:32:50] When moving a piano, I reach for the bench. [00:32:58] Sleep is the cousin of death. [00:33:04] All toasters toast toast. [00:33:05] legoktm: i'll forward you the thread i mentioned earlier [00:33:12] thanks [00:34:56] legoktm: i don't think i have your e-mail address [00:35:07] legoktm@wikimedia.org ;) [00:35:33] Touché. [00:41:39] sent [00:42:23] ping krenair too if you do something with it [00:42:45] https://www.mediawiki.org/wiki/Requests_for_comment/Publishing_the_RecentChanges_feed [00:43:36] Thanks. [00:43:40] not just scale, harden [00:43:58] preventing abuse, etc. [00:43:58] Is that a technical term? [00:44:14] I usually see "secure" used in that context. [00:44:18] Or "rate-limit." [00:45:07] https://en.wikipedia.org/wiki/Hardening_(computing) [00:46:21] websocket used harden! [00:46:24] Metapod, use harden! [00:46:34] ori: Fair enough. [00:46:43] * closedmouth glares at Reedy [00:47:17] (i just paralysed you) [00:47:31] it was super-effective [00:51:18] legoktm: "Basically what we need" isn't terribly useful; i'd summarize the points from the email [00:51:59] yeah, planning on doing that [00:52:33] I can't be fucked to write an RFC, but I have no problem driving from the backseat, mind. [00:52:37] ;-) [00:54:04] i specialize in backseats and backdoors [00:54:18] So many #euphemisii. [01:03:48] This e-mail is bizarre to read. [01:03:57] Because it seems like it would go in reverse chronological order. [01:03:58] But it doesn't. [06:26:16] Is there a reason for none all but the latest 3 or so data dumps for a wiki available? [06:28:27] Space, I think. [06:28:32] There might be mirrors. [07:16:45] hello [07:17:00] on Commons, I got [07:17:01] A database query error has occurred. This may indicate a bug in the software. [07:17:02] Function: EditPage::getLastDelete [07:17:02] Error: 0 [07:17:14] https://commons.wikimedia.org/w/index.php?title=Commons:Undeletion_requests/Current_requests&action=edit§ion=8 [07:38:58] ok now [10:25:25] guillom: we have only 3 translatable docs tasks open O_o [11:12:01] Nemo_bis: yes, the kids have been working like crazy. I'm not sure if Quim wants more tasks created; perhaps you could ask him? If he's ok with it, I can create more. [12:44:20] [[Tech]]; QuiteUnusual; /* Why can't the abuse filter log be viewed.... */; https://meta.wikimedia.org/w/index.php?diff=6920672&oldid=6885287&rcid=4788475 [15:29:33] I'm not sure if this is the right forum for this, but I happened upon an interesting revision history on English WP: http://en.wikipedia.org/wiki/Esoteric_programming_language [15:30:01] At least to me the vandalism is still visible, but in the history I can see ClueBot reverting it correctly. [15:31:40] Tried purging or null editing the page? [15:31:51] Yup [15:31:53] Purge fixed it [15:34:00] https://en.wikipedia.org/w/index.php?title=Esoteric_programming_language&oldid=588811801 is somewhat amusing [15:34:14] Reedy: all right, I didn't know about purging or null editing :) [15:35:10] Reedy: this is supposedly happenning often-ish these days :/ [15:35:24] eml_: https://www.mediawiki.org/wiki/Manual:Purge [15:35:28] MatmaRex: On reverts? [15:35:35] yup, interesting read [15:35:47] Reedy: probably on various edits, but vandalism reverts get noticed most often [15:36:03] people complain on VPT [15:36:28] Is there a bug for it? :P [15:37:28] who knows [15:37:43] there was at least one that was marked as FIXED some time ago, i think [15:38:18] Reedy: see #Radiohead on VPT, for example [15:38:52] (actually, that might not be it) [15:56:54] Is there someone here who knows the Bookmanager API ? [18:08:11] hello [18:08:17] https://commons.wikimedia.org/w/index.php?title=File:Axizz1.jpg&action=delete&wpReason=%5B%5BCommons%3ADeletion+requests%2FFile%3AAxizz1.jpg%5D%5D%3A+This+band+image+is+in+use+on+other+websites%2C+see+%5Bhttps%3A%2F%2Fsoundcloud.com%2Fdj-axizz+here%5D+for+example.++Possible+%5B%5BCOM%3ACOPYVIO%5D%5D. [18:08:17] damn it [18:08:19] Yup [18:08:22] We know [18:08:26] [a527f119] 2014-01-02 18:07:54: Fatal exception of type MWException [18:08:36] lol, ok [18:08:46] We've known for 25 minutes or so ;) [18:09:32] I was wondering if it's me only... [18:09:59] nope [18:10:08] though I'm surprised how long it took for people to notice [18:10:14] notice and report [18:11:15] people expect it to be temporary [18:19:07] and it doesn't happen every time [18:23:05] what's happening with commons? [18:23:06] yannf: Depends if you hit the "right" server [18:26:42] You broke Commons! [18:27:26] i did [18:27:29] I have a habit of doing it [18:30:37] Something wrong in commons? [18:30:46] indeed [18:31:39] http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&x=0.5&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1 [18:31:45] Reedy: I have a different error message on Commons, if that helps. [18:31:53] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [18:32:06] We don't need that to tell us what's wrong ;) [18:32:18] Holy shit 500 million [18:32:23] :? [18:32:30] no, that's milli [18:32:38] Okay. [18:32:38] :P [18:32:40] > 0 [18:33:14] Bsadowski1, it's not yet "fatal" ;) [18:33:27] yeah, I know, just letting you know in case the message itself is helpful [18:41:13] [86f366d4] 2014-01-02 18:37:42: Fatal exception of type MWException [18:41:24] anyone else getting these? (commons) [18:41:34] Everyone, dschwen [18:41:35] dschwen: Known [18:41:39] dschwen: Thousands of people, I expect [18:42:16] hope that's easier to relate to the actual issue for non-tech. [18:42:38] nice clear error btw, good luck for the tech's [18:43:08] akoopal: They of course have full stack traces [18:43:33] yeah of course [18:43:44] first time I see it break this hard [18:44:05] You must be new here. [18:44:11] Seen worse back in '06-'08. [18:44:33] akoopal: :-P [18:44:42] This one time.. [18:45:03] we lost edits on eswiki(?) because the MySQL master ran ouf of disk [18:45:18] + space [18:46:32] started 2008 :-) [18:52:15] O_O [18:56:21] btw we also have PHP fatal error in /usr/local/apache/common-local/php-1.23wmf8/extensions/Translate/tag/TranslatablePage.php line 219: Call to a member function getMessage() on a non-object https://www.mediawiki.org/wiki/Help:Templates/de [18:56:45] Nemo_bis: Relate [18:56:46] d [18:57:15] yes, that's what I meant [18:57:30] error rate increases, sob [19:07:52] commons down? [19:08:25] bennylin: Wikimedia Commons, #Wikidata and all #Wikivoyage projects have been down since around 17:45 UTC due to localization issues. Fixes upcoming. [19:08:32] <\pi{r^2}> hi bennylin [19:08:37] thx for confirm [19:08:43] hi, pir2 [19:29:12] hi, there is a problem in commons [19:29:49] Commons is working again here. [19:29:57] Halfff [19:30:05] Seems up now. [19:30:24] every time I try to load page says "[43406dbb] 2014-01-02 19:23:25: Fatal exception of type MWException" [19:30:36] oh, yes, just right now is working [19:30:59] mmm, no, it's nots [19:31:09] twkozlowski: Is everything working again? [19:31:11] https://commons.wikimedia.org/wiki/Special:Watchlist [19:31:14] ganimedes: it's hit and miss right now, and yeah, we know :) [19:31:19] Oh, it’s down again. [19:31:35] it depends on which server you hit. [19:31:44] so: things are wonky still, yes, working on it :) [19:31:48] I guess you are advice, just want to know what is going on [19:32:07] :) [19:34:46] localization file breakage [19:37:02] sjoerddebruin: Yeah, mostly up [19:37:14] :) [19:43:58] (y) (_y_) (y) [19:44:44] separatist locals? >:> [19:45:43] Huh? [19:46:42] <\pi{r^2}> Separatist locals. [19:47:47] i read "local.... breakage" above... [20:33:21] [[Tech]]; PiRSquared17; /* Thanking on Login Wiki */ +; https://meta.wikimedia.org/w/index.php?diff=6925465&oldid=6920672&rcid=4789633 [20:35:47] uh, I have 96 unread messages on strategy wiki [20:35:59] "new messages", rather [20:40:34] sob, meta read only [20:42:04] what happens to db1007, db1028 [20:43:13] now lag is 0 but keeps being read only [20:43:20] coffee time I guess :) [20:44:00] apergos: ^ db1007 one of yours as a snapshot host? [20:46:48] grrargh, database locked [20:47:35] eh? [20:47:54] apergos: See operations [20:48:45] what do you mean 'one of mine'? [20:49:20] Reedy: [20:49:38] snapshot host [20:49:44] isn't that used for dumps? [20:51:52] why can i not edit commons ("database locked") but others can, as it seems from recent changes? [20:52:46] Ctrl+R [20:52:51] same on Meta even though replag is 0, is that stored in memcached or something? [20:53:04] centralauth is on s7 [20:53:11] I suspect that's why it's affecting wider [20:53:57] actually, dbtree says "not reporting or replicting" [20:54:11] for all s7 slaves O_o [20:54:38] db1007? no, it's not tagges for dumps [20:54:49] 'db1007' => 0, # snapshot, vslow [20:54:56] it is in db-eqiad [20:55:14] Either s7 database issues [20:55:40] Nemo_bis: [20:43:38] PROBLEM - MySQL Slave Running on db1007 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 39425319 for key PRIMARY on query. Defaul [20:55:42] that's replication related [20:55:44] there's only one host tagged for dumps right now and that's db1043 and only if en wp is running that branch [20:55:52] not just s7 [20:55:56] * aude thinks [20:56:01] aude: centralauth is ons7 [20:56:05] can likely screw any wiki up [20:56:11] ok [20:57:36] apergos: Can you have a look at the s7 master? db1039 [20:58:08] huwiki and viwiki currently being dumped are both s7 http://dumps.wikimedia.org/backup-index.html [20:59:11] at this time of night my brain is mostly gone. but I will at least see if there is anything that leaps out at me [21:02:05] The server for https://no.wikipedia.org has been reporting that it is locked to catch up with the master for some time now, usually this should go pretty fast but now it seems to be stuck? [21:02:20] jeblad: master is unhappy [21:02:21] we know [21:02:34] hi jeblad ! [21:02:37] hi aude ! [21:02:42] hey jeblad :) [21:02:52] Who's running a schema change? [21:02:56] https://twitter.com/twkozlowski/status/418849321681821696 [21:02:59] jeblad: ^^ [21:03:01] Reedy: Try to spank the master.. [21:03:31] does this mean something? https://ganglia.wikimedia.org/latest/graph_all_periods.php?title=mysql_innodb_transactions_unpurged&mreg[]=^mysql_innodb_transactions_unpurged%24&hreg[]=db10%2807|24|28|41%29&aggregate=1&hl=db1041.eqiad.wmnet|MySQL%20eqiad,db1024.eqiad.wmnet|MySQL%20eqiad,db1007.eqiad.wmnet|MySQL%20eqiad,db1028.eqiad.wmnet|MySQL%20eqiad [21:03:37] hi hoo twkozlowski etc etc.. [21:03:59] ...and happy new year to everybody! [21:04:11] Nemo_bis: seems someone is running a schema change [21:04:25] sob sob [21:04:34] DROP TABLE revision; -- Dropping old clutter [21:04:59] no worries, it's still just 1.5M compared to a 20M peak in March :) [21:05:28] whatever mysql_innodb_transactions_unpurged actually entails [21:06:09] a !log is in order [21:07:00] some people are complaining they cant edit. is this known already? [21:07:01] !log s7 is unhappy [21:07:08] Yes [21:07:10] ok [21:07:37] morebots feels asocial today [21:07:53] soapaway: in general, please refer to channel topic in case of outages, etc. [21:08:00] database locked… i can't save my edits [21:08:12] Database replication issues on some wikis, sorry! [21:08:25] User::saveOptions [21:08:58] yeah i know i was just asking because it doesnt say anything in the topic about it being down [21:09:08] yes it does [21:09:10] though i see now 's7 database issues (including centralauth)' [21:09:17] i woukdnt have known that that means it's downm [21:09:24] it's not down. [21:09:25] no idea what it's about [21:09:30] you just can't edit the site. [21:09:39] editing works fine for me [21:09:45] it seems like recent changes is about 1/3 of normal [21:09:57] in fact [21:10:01] it seems to maybe be fully back now [21:10:18] fixed on ca.wiki! [21:10:22] works now on commons [21:10:25] im trying to get the ones who were saying it's dwon to try their edits again [21:10:27] thanks to whoever fixed it [21:10:40] !blame [21:10:40] Whatever happened, it's Domas' fault. For everything else, there's `git blame`. [21:10:46] ok cool [21:10:53] :) [21:10:54] INSERT /* User::saveOptions xxxxx */ IGNORE INTO `user_properties` (up_user,up_property,up_value) VALUES ( stuff,'rememberpassword','1'),(stuff,'echo-subscriptions-web-reverted',''),(stuff,'echo-subscriptions-email-article-linked','1') etc, that's the only thing I see on the master [21:11:06] WHATSUP HOMIES [21:11:07] holding locks [21:11:52] or held one anyways [21:12:27] works again for me [21:12:54] yup, I know who fixed it [21:13:07] :-) [21:13:53] Thanks! =) [21:17:32] [[Tech]]; Nemo bis; /* Thanking on Login Wiki */ +re; https://meta.wikimedia.org/w/index.php?diff=6925678&oldid=6925465&rcid=4789687 [21:26:49] <\pi{r^2}> hi wizardist [21:27:06] arigato [21:30:57] one user at be.wikipedia.org has DB-related issue: "The database has been automatically locked while the slave database servers catch up to the master". I can read the status line of the channel but AFAIK bewiki belongs to s3, not s7 [21:31:58] two other users (wiki sysops) incl. me couldn't reproduce it, but the user insists the issue is persistant [21:32:09] wizardist: not anymore [21:32:15] twkozlowski: s7? [21:32:36] wizardist: make them check again, there shouldn't be any issues at this time [21:32:41] you mean bewiki lives at s7? [21:32:46] domas fixed Things. [21:32:55] s7 has centralauth [21:32:56] yeah, I've already asked them. waiting for response. [21:32:58] anything can get upset [21:33:04] reasonable [21:33:27] <\pi{r^2}> Is bewiki s3? [21:33:39] https://noc.wikimedia.org go check [21:33:44] Yup [21:33:58] yep, it's not listed => going to DEFAULT [21:34:12] which is s3 to be precise [21:34:16] <\pi{r^2}> Yep. [21:34:25] ok, thanks for input [21:34:36] meh, probably inherited s7 data somehow [21:34:39] https://noc.wikimedia.org/conf/highlight.php?file=s3.dblist [21:34:49] domas: centralauth tables are on s7 [21:34:53] so any wiki can try and use them [21:34:55] yeah, probably [21:35:23] remark: the issue arose when the user tried to create a new page. just in case it could depend on smth [21:35:41] create -> save [21:56:14] Is it possible to check the origin of traffic on a specific article? [21:56:41] Seems like there is some bot trffi that is completly insane.. [21:57:08] I think I know who it is but I would like some verification if possible [21:59:11]