[00:22:11] werdna: is echo running? [00:22:22] AaronSchulz: no [00:22:25] we pulled it [00:42:59] !log asher synchronized wmf-config/db.php 'adding db63 to s1 at a low weight' [00:43:07] Logged the message, Master [00:43:35] binasher: heya asher [00:43:44] hey [00:43:58] have you seen TCP Fast Open? :) [00:44:06] LWN has a nice article about it. [00:45:14] no, looks neat [00:45:39] i've followed some of google's tcp / protocol improvements but not this one [00:46:13] I'm not terribly fond of the way the socket api is changed [00:46:34] but the end results sounds very nice to have [00:47:17] oh and btw, I think our SSL stuff can be optimized a bit wrt RTT [00:47:18] !log reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php [00:47:26] Logged the message, Master [00:54:53] are you still investigating spdy? [00:55:32] !log asher synchronized wmf-config/db.php 'moving watchlist/special contrib queries to db63, preparing to decom db12' [00:55:41] Logged the message, Master [01:00:23] I'm not really "investigating" it [01:00:45] I'm reading about it and keep myself up to date [01:01:06] but I'm busy with other stuff to pursue this project for us [01:01:46] but if you're interested in seeing this, I'd certainly attempt to join you ;-) [01:02:10] ("attempt" since my hands are full already) [01:03:58] binasher_: lost the backlog I presume? :) [01:05:00] a little bit [01:05:26] 04:00 < paravoid> I'm not really "investigating" it [01:05:26] 04:00 < paravoid> I'm reading about it and keep myself up to date [01:05:26] 04:01 < paravoid> but I'm busy with other stuff to pursue this project for us [01:05:29] 04:01 < paravoid> but if you're interested in seeing this, I'd certainly attempt to join you ;-) [01:05:32] 04:02 < paravoid> ("attempt" since my hands are full already) [01:07:37] binasher: btw, do you know which server andrew bogott used for the ceph perf benchmarks? [01:07:53] virt1004 has a failed disk... [01:08:41] virt1001-1003 i think, i didn't look at his setup though [01:08:51] but he was aware of the failed disk in 1004 [01:10:00] ah, okay [01:11:03] TimStarling: LWN has an article about the "leap second of doom" (its reoccurence this month specifically) [01:11:15] not much new info other than links to the ntp thread [01:11:49] the object store is supposed to be a little more production worthy than the network block store [01:12:04] eh? [01:12:46] there's rados (the object store), rbd (block device over rados, either in kernel- or userspace) and cephfs (posix fs over rados) [01:12:54] all of them together being called ceph, confusingly [01:13:27] I think none of them are production worthy but I'd say the same for 100% of our current stack [01:13:49] (openstack nova/keystone/whatever, gluster, the buggy kvm that we run etc.) [01:16:48] !log reedy synchronized php-1.20wmf8/extensions/SubPageList3/SubPageList3.php [01:16:58] Logged the message, Master [01:26:20] !log reedy synchronized php-1.20wmf8/extensions/SubPageList3/ [01:26:28] Logged the message, Master [01:27:13] paravoid: do you have a DD subscription to LWN? [01:28:29] yes [01:29:21] thanks to HP [01:30:25] oh, i didn't know that came from them [01:30:32] * jeremyb cheers bdale [01:33:42] paravoid: speaking of production worthy, supposedly rackspace is using more openstack than they were (I guess they were mostly swift before and now they're nova too?) [01:34:13] was some kind of opt-in period i think and now it's available for anyone [01:34:32] (i think) [02:01:09] !log reedy synchronized php-1.20wmf8/extensions/E3Experiments/ [02:01:17] Logged the message, Master [02:27:17] !log LocalisationUpdate completed (1.20wmf8) at Thu Aug 2 02:27:17 UTC 2012 [02:27:25] Logged the message, Master [04:45:25] !log tstarling synchronized php-1.20wmf8/skins/common/commonElements.css [04:45:34] Logged the message, Master [09:22:31] Hallo. Is there any chance to resolve https://bugzilla.wikimedia.org/show_bug.cgi?id=38946 in a day or two? It shouldn't be hard, but maybe I'm missing something important. [09:27:22] I got a similar request with https://bugzilla.wikimedia.org/38299 / https://gerrit.wikimedia.org/r/#/c/15550/ [09:27:50] aharoni: isn't there an Ubuntu package that provides the fonts ? [09:28:02] Not all of them. [09:28:16] $ apt-cache search culmus [09:28:16] culmus - Type1 Hebrew Fonts for X11 [09:28:17] culmus-fancy - Type1 Fancy Hebrew Fonts for X11 [09:28:33] Yes, but some of them are not in that package. [09:40:47] aharoni: Lucid has culmus 0.101 [09:40:55] Ubuntu Precise has 0.120 [09:41:00] and Quantal 0.121 [09:41:10] maybe it one of the package can be back ported to lucid [09:42:26] maybe one of the packages can be backported to Lucid [12:49:28] going to deploy some important fixes for Translate [13:01:58] domas: do you have any performance tips on how to import the wikimedia dumps into a mysql db? atm i'm just interested in the page.sql.gz, and the pagelinks.sql.gz for some research, so there won't be any concurrent writes on those tables after importing. seems myisam is way faster for importing than innodb, but it still takes several days whenever there's a new dump available... [13:02:17] someone pointed me to your myloader, but it seems it expects a directory in a certain format, not a single sql dump :-/ [13:02:38] jorn: bump buffer pool / transaction log sizes [13:02:42] disable transaction log flushing [13:02:50] yay, 1000x faster data loads [13:03:06] alternative is loading into table with no secondary indexes then recreating indexes [13:05:26] !log nikerabbit synchronized php-1.20wmf8/extensions/Translate/specials/ 'Translate bug fixes' [13:05:35] Logged the message, Master [13:09:04] thanks, will check those again (i remember i already bumped the key_buffer to 2G and increased some other buffers), but the choice of myisam over innodb is ok in my case? [13:14:22] ^demon: someone sent an Intent to Package to Debian about apache-sshd, noting how it's a dependency for Gerrit [13:14:29] ^demon: probably he's packaging Gerrit as well [13:14:50] <^demon> That'd be cool, if it happens :) [13:17:50] jorn: if you're not going to use prod workload, myisam may work fine [13:17:53] heh [14:16:15] The new HTTPS banner is *very* annoying [14:46:44] * jeremyb wonders what banner that was a reference to [15:03:50] paravoid: i'm not seeing this apache-ssh [15:04:08] #683639 [15:04:15] there is however a gerrit RFP [15:04:17] * jeremyb looks [15:04:27] !debbug [15:04:40] !debbug is http://bugs.debian.org/$1 [15:04:40] Key was added [15:04:44] also #683642, gwt-maven-plugin [15:05:00] !debbug 589436 [15:05:00] http://bugs.debian.org/589436 [15:05:12] says that it's packaged as a dep fro gerrit [15:05:31] right, but the other two were ITPed today [15:05:36] so I guess Thomas is doing some progress [15:06:33] hello, jeremyb? [15:06:37] I wasn't in the channel [15:06:45] ok? [15:07:01] paravoid: ohhh, i guess wnpp hasn't caught up [15:07:22] also, 681918 [15:08:25] cashews: so what banner? [15:08:39] the blue and black ones - i'll ?uselang=qqx it [15:09:14] I�m getting some Wikimedia errors at nl.wikipedia [15:09:29] anyone have a minute to write a quick js tool? [15:09:29] Anyone knows if there�s a problem? [15:09:49] Wiki13: there's no known problem afaik, could you give a bit more info? [15:09:57] Wiki13: what errors, which page [15:10:04] cashews: qqq ? [15:10:20] gimme a sec [15:10:52] Never mind, it stopped ging errors already [15:10:59] giving* [15:11:07] Wiki13: nevermind, we're getting tons of alerts right now :> [15:11:08] gah, matanya was here... was looking for you! [15:11:16] whoa [15:11:23] uh [15:11:33] the mobile site badly broken right now [15:11:54] Nikerabbit: did you see nagios? ;) [15:12:03] i think mobile is the least of our worries [15:12:10] Centralnotice-template-B12 071912 NofaceTopBlueFactsask [15:12:37] anyone here who might know how this is possible? there are 2 pages included in the redirect table which don't have the page.page_is_redirect flag set: http://p.defau.lt/?7lWkA9HvViIYdZlD4yVm_w [15:12:41] jeremyb: no I did not see anything, status says up [15:13:19] Nikerabbit: well just read in here since 15:08 UTC then [15:13:24] (5 mins) [15:13:39] jorn: later, after outage is sorted [15:13:54] still both pages redirect correctly: http://en.wikipedia.org/wiki/Archytas_apcifer and http://en.wikipedia.org/wiki/The_Ministry_of_the_Heavenly_Vessel [15:14:00] jeremyb: ok, sorry [15:17:49] works better already [15:18:54] cashews: i don't think that identifies an individual banner? https://meta.wikimedia.org/w/index.php?title=Special:NoticeTemplate&offset=&limit=100&tplsearchkey=B12 [15:24:07] jorn: maybe we can get someone to run the same query on toolserver or prod? [15:24:40] jeremyb: uhm yes, just know that my articles is the page table restricted to namespace 0 [15:25:15] and i'm working on the july dump [15:25:18] * jeremyb wonders if Brooke is around? [15:25:43] jeremyb: I have toolserver access, if you need me to run a query. [15:26:02] Matthew_: http://p.defau.lt/?7lWkA9HvViIYdZlD4yVm_w [15:27:19] * Matthew_ looks [15:28:26] jeremyb: Error 1146 [15:28:30] and there's another weirdness: http://p.defau.lt/?NlZcwR6IT3cLwhzEt25N_g http://en.wikipedia.org/wiki/Atlantic_Swordfish and http://en.wikipedia.org/wiki/Speckled_garden_eel are marked as page_is_redirect but don't appear in the redirect table. even though that might be explained here: http://www.mediawiki.org/wiki/Manual:Redirect_table (see the NOTE as of Aug 2007) [15:29:12] Matthew_: as i said you need to substitute "articles" with the page table restricted to page_namespace 0 [15:29:37] jorn: LOL, oops. I missed that. [15:31:59] Darn, gateway timeout. [15:33:59] Matthew_: i re-ran it with the page table: http://p.defau.lt/?smGJYbXXlqwLPhreqbe8VA [15:35:27] Matthew_: and the second one: http://p.defau.lt/?pwJZMIVtTGGZYb69o8YWxg [15:39:54] jorn: OK, I got 21 rows on my first set query. Running the second now (for some reason, my queries are lagging significantly) [15:41:13] sure it's lagging? i mean that are a hell lot of rows you're joining there... [15:42:13] but anyhow, my original question was more am i right that it's weird or am i missing something? [15:42:26] Maybe, but "23 rows in set (0.00 sec)" vs "21 rows in set (1 min 5.15 sec)" is a little off. [15:42:44] Matthew_: nah, that was cached [15:42:54] ah, that would explain it. [15:43:07] i lost the output of the first and re-ran ;) [15:43:26] (lost in screen's paging somewhere) [15:43:44] Oh, OK. That would explain it then ;) My second one is running now. [15:45:16] I got two rows on the second one. [15:46:44] weird but ok... i mean i'm working on the dump... any in page_namespace 0 in both? [15:47:30] but in general i think this should not happen as it means that either the redirect or the page table is not accurate [15:47:52] i forget who usually looks at this stuff... Reedy ? [15:47:58] certainly Brooke [15:50:15] jorn: One second, I have to sort the data. [15:50:30] ? [15:51:16] it's just 23 rows, just look at it ;) [15:51:55] 25 with the 2 others... [15:52:31] jorn: Meh, my client wonked up the copy and paste, it had random newlines. Anyway, there are none in page_namespace 0 [16:47:32] jeremyb, Matthew_: i also observe a weird disagreement in the pagelinks table: http://p.defau.lt/?1I18QznyfSY5PRpk2dGvtw so there are pagelinks by pages which are just redirects to articles other than the redirect target? [16:54:23] That is wierd... [17:04:26] Matthew_: i'm currently running a count query on that but that might take some time as it's 670M rows... [17:05:06] jeremyb: any idea if i should report these in bugzilla? [17:05:46] even though i actually have the feeling that bugzilla is more like a write only memory... whenever i report no one cares :-/ [17:12:15] haha [17:12:21] jorn: i think that's not true [17:12:35] jorn: but maybe first let's ask someone that i think does care [17:12:47] idk when they'll respond though... [17:13:06] Brooke is one. the other that i'm thinking of i'm not sure. Krinkle ? Reedy ? [17:13:53] alright, no problem... [17:14:07] i just gtg, but will be back tomorrow [17:14:23] jorn: sure, we have the pastes ;) [17:14:32] jorn: what's your bugzilla account? [17:15:39] https://bugzilla.wikimedia.org/show_bug.cgi?id=38604 [17:15:53] don't want to paste the mail address here [17:17:49] but the write only memory comment was more a general accusation of bugzilla systems... at some point you're just so overflooded with requests that it's impossible to even read them :-/ [17:21:07] heh [17:21:46] jorn: http://hire.jobvite.com/Jobvite/Job.aspx?j=op0lWfwq&c=qSa9VfwQ [17:30:05] heh, thanks will come back to that after the phd ;) currently i'm trying to compare the page impressions to the wikipedia internal pagerank and trying to approximate how many users follow a link from one page to another... [17:30:10] anyhow, gtg, bus, byw [18:35:53] !log Stealing E2's deployment window to have another crack at Echo [18:36:02] Logged the message, junior [18:37:27] ^demon:around? [18:37:49] werdna: There's an echo deployment? [18:37:50] <^demon> What's up? [18:38:06] ^demon: Can you add me to the wmf-deployment group on gerrit? [18:38:09] I never got added [18:38:49] !log Turned Echo back on on testwiki [18:38:57] Logged the message, junior [18:39:07] <^demon> werdna: Doned. [18:39:16] :D [18:40:06] Oh, just testwiki. Right :) [18:44:17] Jarry1250: can I get your thoughts on http://en.wikipedia.org/wiki/Wikipedia:VPT#A_question_concerning_scripting_and_protection [18:44:43] Betacommand: In terms of how to do it? [18:44:51] yeah [18:44:56] looking for a js tool [18:45:07] You'd post to the API. [18:45:36] Jarry1250: my JS skills are about the same as my Klingon skills [18:46:07] Betacommand: Ah, yes, well I can't code in Pascal either :P [18:46:11] Let me think. [18:46:27] Protect will require a token, so it's two requests. [18:49:58] Betacommand: Will get back to you, two secs [18:50:04] Jarry1250: np [18:55:38] ^demon: Know anything about job runners? [18:55:51] <^demon> Nope. [18:57:04] Duhh [18:57:07] They run jobs [18:57:43] really? how fast? [18:58:06] Reedy: I'm finding that if EchoNotificationController::notify() is called from a job runner, no notifications are distributed, but it works if I run it manually [18:58:25] I'm wondering if there is an error log for job runners? [18:59:21] /h/w/log/jobqueue [18:59:25] werdna: there is a config setting about running jobs [18:59:35] nope, all out of date [19:00:00] lol, 8GB of out of date job queue logs [19:00:59] hmmm [19:01:31] werdna: $wgJobRunRate [19:01:34] does it work locally via jobs? just not on WMF [19:01:41] Betacommand: we don't use that on the cluster [19:01:43] Reedy: yeah [19:01:48] works fine via jobs locally [19:02:16] oh [19:02:16] maybe I can try running runJobs.php [19:02:17] -rw-r--r-- 1 999 root 64890728 2012-08-02 19:02 runJobs.log [19:03:27] testwikis job table is empty [19:03:32] "Need expert advice for your website? ..I wanted to know if you had interest in getting a professional website for only 9.95 per month with no setup fee? .." --> "Are you sure you want to offer to rebuild Wikipedia for 10 bucks? :)" [19:03:49] no mention of "Echo" in the last 10k lines of runJobs.php [19:04:43] Reedy: yeahh right now I'm not even seeing this stuff hit the job table [19:05:26] grrrr [19:05:29] That sounds suspect [19:05:43] it was yesterday… [19:18:07] Reedy: this is weird [19:18:15] AaronSchulz: you about? [19:18:44] hmm [19:19:53] AaronSchulz: could there be something wrong with the testwiki job runners? [19:19:54] http://p.defau.lt/?4AYBBfpEGbA4iao6nQJBaQ [19:19:59] but the job is gone instantly [19:20:02] and nothing seems to happen [19:32:33] werdna: doesn't testwiki use the same pool of runners? [19:32:46] Reedy: AaronSchulz suggested it might be more slave lag [19:32:50] the job is done before the event is propagated [19:33:00] lol [19:33:12] sounds possible [19:34:04] I tried a fix for that but it's still broken [19:34:15] What did you try? [19:35:12] add loads of wfWaitForSlaves(); ;) [19:37:32] werdna, I noticed the same problem on testwiki 2 days ago [19:37:47] Reedy: read from master :) [19:38:20] just reading from masters is not enough [19:38:30] * MaxSem suggests also killing the slaves [19:38:47] Slavery is illegal [19:39:35] ...said the herder of a huge cluster full o'slaves [19:39:43] ;) [19:40:40] seriously though, jobs on testwiki indeed look borked [19:40:50] werdna: try test2wiki [19:41:00] it's more production-y [19:41:36] let me see [19:43:11] !log Deploying Echo to test2wiki instead [19:43:19] Logged the message, junior [19:43:43] !log andrew synchronized php-1.20wmf8/extensions/LiquidThreads [19:43:45] did you do the tables already? [19:43:52] Logged the message, Master [19:43:57] Reedy: yep [19:44:07] !log andrew synchronized php-1.20wmf8/extensions/Echo [19:44:15] Logged the message, Master [19:44:29] !log andrew synchronized wmf-config/InitialiseSettings.php [19:44:37] Logged the message, Master [19:45:28] ha! [19:45:29] it works [19:45:50] fuck testwiki [19:46:10] seriously [19:47:44] did you mean: test fuckwiki? [19:47:55] preilly, we had the same problem on Tuesday^^ [19:54:18] !log andrew synchronized php-1.20wmf8/extensions/LiquidThreads [19:54:27] Logged the message, Master [19:55:27] !log andrew synchronized php-1.20wmf8/extensions/Echo [19:55:35] Logged the message, Master [19:55:46] !log andrew synchronized wmf-config/CommonSettings.php [19:55:54] Logged the message, Master [19:56:06] !log andrew synchronized wmf-config/InitialiseSettings.php [19:56:11] ori-l: you need to deploy any E3Experiment updates or config changes today? [19:56:15] Logged the message, Master [19:56:26] kaldari: nope, check yr mail :) [19:56:36] !log deployed Echo to mediawiki.org [19:56:36] ah :) [19:56:44] Logged the message, junior [19:56:51] kaldari: thanks for asking, though [20:01:41] ugh! it works on test2wiki but not mediawiki [20:01:42] grrrrr [20:02:28] !log andrew synchronized php-1.20wmf8/extensions/LiquidThreads [20:02:36] Logged the message, Master [20:04:29] !log Rolled back LQT updates, they seem to break other notifications. We'll figure out the bugs with that in another window. [20:04:37] Logged the message, junior [20:06:53] !log Leaving Echo deployment there for now. [20:07:01] Logged the message, junior [20:12:30] !log asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent = true as an experiment' [20:12:38] Logged the message, Master [20:30:13] !log asher synchronized wmf-config/mc.php 'setting wgMemCachedPersistent back to false, no change observed to current issue' [20:30:22] Logged the message, Master [20:34:28] !log andrew synchronized php-1.20wmf8/extensions/Echo/modules/base/ext.echo.base.css [20:34:36] Logged the message, Master [20:46:26] * AaronSchulz hears wtfs coming from asher's direction [21:10:48] Okay who broke the wiki [21:11:06] 502 Bad Gateway [21:11:07] 502 Bad Gateway when viewing any page [21:11:12] evening guys, you might wanna take a look at enwp - 502 Bad gateway for some, I simply can't connect at all, not even getting an error [21:11:17] yep, on all wikis. not just enwiki [21:11:23] but enwikinews, uk.wikimedia.org, commons etc. [21:11:27] just one of firefox's standard "can't connect to en.wikipedia.org" [21:12:01] * Headbomb kicks the servers [21:12:21] Danny_B|backup: It's not some wikis, it's all wikis. [21:12:38] mysterytrey in #wikipedia-en is saying secure is live for en users [21:13:20] What happened to the error message telling people to flood this channel? [21:13:32] hi [21:13:33] well, 502 Bad Gateway [21:13:35] is it down? [21:13:38] Danny_B|backup: Only some wikis? [21:13:46] every damn wiki I try is down [21:13:52] pl.wikipedia down for me with 502 bad gateway error. [21:13:53] wikitech.wikimedia.org isn't. :-) [21:14:07] ha, someone has already blamed me on twitter [21:14:13] But every cluster wiki appears either 502 or no route. [21:14:16] wikitech isn't hosted on our servers :) [21:14:18] this is what you get when you take a wikibreak. ;-) [21:14:21] what's the homepage to route in to en.wikipedia via secure.wikimedia.org [21:14:33] BarkingFish: don't use secure [21:14:35] LeslieCarr: Indeed. :-) [21:14:37] Dispenser, it's just common practice that people go to go here if the wiki is down. [21:14:40] and don't recommend that people do so [21:14:45] -go [21:14:53] Ryan_Lane, mysterytrey reckons you can get into it that way [21:15:12] we should just delete secure, blecch [21:15:16] BarkingFish: only if you want to make our lives harder [21:16:05] Bsadowski1: It was tongue in cheek. The green screen of death had a webchat link here that all the noobs clicked [21:16:12] :P [21:16:40] Oh gawd. [21:16:41] "Bad Gateway" is new [21:17:08] Platonides, Bad Gateway only on HTTPS [21:17:09] Ryan_Lane, I've passed that on to #wikipedia-en [21:17:51] oh yes, can't even connect to the LVS on http [21:18:07] good luck, guys, with the wiki servers (?) problem [21:19:11] tyw7 is asking to be unblocked in here [21:19:36] gn8 folks [21:19:45] who's tyw7 ? [21:19:49] no clue [21:19:54] he's banned apparently, though [21:19:54] thanks guys [21:20:10] LeslieCarr: from before your time in this channel :D [21:20:14] Oh, I remember him. [21:20:22] why was he banned? [21:20:49] no clue. for being repeatedly annoying i believe [21:21:00] back in business? [21:21:37] probably brion used the hammer during some down time of the site :D [21:21:38] Ryan: Also retired & topic banned on en.wp [21:21:46] WilliamH_UK: secure is running if you need to check something [21:21:58] well there's less squids dead now, the number is going down :) [21:21:59] both are working for me [21:22:06] Some people are reporting that they can get in [21:22:32] BarkingFish: To the mobile site, or the desktop one? [21:22:42] desktop [21:22:51] both seem ok now [21:23:19] from #wikipedia, James_F - BarkingFish: site is back [21:23:34] i assume he means desktop [21:23:37] i'll check [21:23:49] https pl.wikipedia up [21:24:02] so, yeah, what caused that then? [21:24:26] yes, let the public shaming begin !!! ;) [21:24:29] tommorris: We're trying to make Twitter look better by having more downtime than them. :-) [21:24:41] tyw7 is saying mobile is up, that's definite [21:24:56] it's up for me in europe as well [21:24:57] and Fluffernutter confirms regular site is live too [21:25:05] (hi) [21:25:10] Ryan_Lane: maybe it was a long time ago? not seeing much in my logs [21:25:11] oops, sorry :) [21:25:16] didn't see you, Fluffernutter :D [21:25:17] * tommorris goes back to his wikibreak. seeya. [21:25:20] (at least through the beginning of 2011 [21:25:21] ) [21:26:57] Ryan_Lane, can you check that ban on tyw7 please? He's saying he still cannot send to the channel here [21:27:07] I don't see him in the banlist [21:27:23] tyw7, I think your ban in #wikimedia-tech has been lifted [21:27:23] still cannot... [21:27:23] * Ryan_Lane removes ban on *!*@wikipedia/tyw7 [21:27:23] [22:26] == Cannot send to channel: #wikimedia-tech [21:27:36] so, if I want to puppetize the deployment of a much smaller mediawiki deployment, should I build my own manifests, or is it feasible to use the wikimedia ones, after heavily suppressing them [21:27:44] if you can find him in the banlist, I'll gladly remove it [21:27:47] tyw7: what's your IP? [21:27:53] jeremyb: do a whois [21:27:56] lol. [21:28:00] Ryan_Lane: huh? [21:28:10] Ryan_Lane: oh [21:28:11] /whois tyw7 [21:28:24] Ryan_Lane: *normally* that doesn't work, tyw7's special ;) [21:28:32] oohhhhhhhhhhhhhhh [21:28:34] he's in the channel [21:28:38] yes [21:28:40] he's not voinced [21:28:42]