[00:16:20] ori-l, agree :) [00:16:26] will be a fun project to change all of it :) [00:16:55] and we really ought to do the generic json config... [00:16:59] one of these days ... [00:53:45] (03PS2) 10Legoktm: Add $wmgCirrusSearchEnablePref, which adds a BetaFeature [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98046 [00:55:18] (03CR) 10Legoktm: "Ok, didn't know that. I've added in another variable to control it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98046 (owner: 10Legoktm) [00:56:40] addshore: ansible is so slow! [00:56:56] addshore: but I'm running it on my computer to a remote host in the US, so... :) [02:10:13] !log LocalisationUpdate completed (1.23wmf4) at Sat Nov 30 02:10:13 UTC 2013 [02:10:30] Logged the message, Master [02:18:36] !log LocalisationUpdate completed (1.23wmf5) at Sat Nov 30 02:18:36 UTC 2013 [02:18:51] Logged the message, Master [02:47:22] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 30 02:47:22 UTC 2013 [02:47:36] Logged the message, Master [03:27:07] Need the go-ahead for a 10k bigdelete\ [03:36:52] PROBLEM - Apache HTTP on mw1201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:52] PROBLEM - Apache HTTP on mw1195 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:02] PROBLEM - Apache HTTP on mw1202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:02] PROBLEM - Apache HTTP on mw1196 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:02] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:02] PROBLEM - Apache HTTP on mw1189 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:02] PROBLEM - Apache HTTP on mw1206 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:02] PROBLEM - Apache HTTP on mw1200 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:02] PROBLEM - Apache HTTP on mw1203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:12] PROBLEM - Apache HTTP on mw1191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:32] PROBLEM - Apache HTTP on mw1120 is CRITICAL: Connection timed out [03:37:32] PROBLEM - Apache HTTP on mw1140 is CRITICAL: Connection timed out [03:37:32] PROBLEM - Apache HTTP on mw1193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:32] PROBLEM - Apache HTTP on mw1197 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:32] PROBLEM - Apache HTTP on mw1190 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:32] PROBLEM - Apache HTTP on mw1192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:32] PROBLEM - Apache HTTP on mw1136 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:33] PROBLEM - Apache HTTP on mw1129 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:33] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:34] PROBLEM - Apache HTTP on mw1199 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:34] PROBLEM - Apache HTTP on mw1208 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:35] PROBLEM - Apache HTTP on mw1207 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:35] PROBLEM - Apache HTTP on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:36] PROBLEM - Apache HTTP on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:36] PROBLEM - Apache HTTP on mw1124 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:37] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:42] PROBLEM - Apache HTTP on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:42] PROBLEM - Apache HTTP on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:42] PROBLEM - Apache HTTP on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:42] PROBLEM - Apache HTTP on mw1205 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:42] PROBLEM - Apache HTTP on mw1204 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:42] PROBLEM - Apache HTTP on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:42] PROBLEM - Apache HTTP on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:43] PROBLEM - Apache HTTP on mw1122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:43] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:44] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:52] PROBLEM - Apache HTTP on mw1117 is CRITICAL: Connection timed out [03:37:52] PROBLEM - Apache HTTP on mw1134 is CRITICAL: Connection timed out [03:37:52] PROBLEM - Apache HTTP on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:02] PROBLEM - Apache HTTP on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:02] PROBLEM - Apache HTTP on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:02] PROBLEM - Apache HTTP on mw1135 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:02] PROBLEM - Apache HTTP on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:02] PROBLEM - Apache HTTP on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:02] PROBLEM - Apache HTTP on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:02] PROBLEM - Apache HTTP on mw1147 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:03] PROBLEM - Apache HTTP on mw1125 is CRITICAL: Connection timed out [03:38:03] PROBLEM - Apache HTTP on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:04] PROBLEM - Apache HTTP on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:04] PROBLEM - Apache HTTP on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:05] PROBLEM - Apache HTTP on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:12] PROBLEM - Apache HTTP on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:12] PROBLEM - Apache HTTP on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:12] PROBLEM - Apache HTTP on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:12] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:22] RECOVERY - Apache HTTP on mw1190 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.184 second response time [03:38:32] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.140 second response time [03:38:32] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [03:39:02] PROBLEM - Apache HTTP on mw1194 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:52] RECOVERY - Apache HTTP on mw1203 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 1.020 second response time [03:40:02] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.679 second response time [03:40:22] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.118 second response time [03:40:32] RECOVERY - Apache HTTP on mw1208 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.971 second response time [03:41:02] RECOVERY - Apache HTTP on mw1189 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.764 second response time [03:41:32] PROBLEM - Apache HTTP on mw1190 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:42:22] RECOVERY - Apache HTTP on mw1190 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.887 second response time [03:42:32] RECOVERY - Apache HTTP on mw1192 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 8.134 second response time [03:42:32] RECOVERY - Apache HTTP on mw1207 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.569 second response time [03:42:42] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.214 second response time [03:42:42] RECOVERY - Apache HTTP on mw1201 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.108 second response time [03:42:52] RECOVERY - Apache HTTP on mw1200 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.194 second response time [03:42:52] RECOVERY - Apache HTTP on mw1138 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.493 second response time [03:42:53] RECOVERY - Apache HTTP on mw1202 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 4.759 second response time [03:43:02] RECOVERY - Apache HTTP on mw1191 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 2.334 second response time [03:43:02] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.211 second response time [03:43:02] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.032 second response time [03:43:22] RECOVERY - Apache HTTP on mw1193 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 1.136 second response time [03:43:22] RECOVERY - Apache HTTP on mw1197 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.042 second response time [03:43:22] RECOVERY - Apache HTTP on mw1199 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.045 second response time [03:43:23] RECOVERY - Apache HTTP on mw1124 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.274 second response time [03:43:23] RECOVERY - Apache HTTP on mw1136 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.323 second response time [03:43:32] RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 1.809 second response time [03:43:32] RECOVERY - Apache HTTP on mw1204 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.039 second response time [03:43:32] RECOVERY - Apache HTTP on mw1116 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.226 second response time [03:43:32] RECOVERY - Apache HTTP on mw1114 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.331 second response time [03:43:32] RECOVERY - Apache HTTP on mw1205 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.783 second response time [03:43:32] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.229 second response time [03:43:42] RECOVERY - Apache HTTP on mw1131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.274 second response time [03:43:42] RECOVERY - Apache HTTP on mw1195 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.108 second response time [03:43:42] RECOVERY - Apache HTTP on mw1128 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.273 second response time [03:43:52] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 5.108 second response time [03:43:52] RECOVERY - Apache HTTP on mw1117 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.295 second response time [03:43:52] RECOVERY - Apache HTTP on mw1194 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.039 second response time [03:43:52] RECOVERY - Apache HTTP on mw1135 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.199 second response time [03:43:52] RECOVERY - Apache HTTP on mw1118 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.223 second response time [03:43:52] RECOVERY - Apache HTTP on mw1145 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.217 second response time [03:43:53] RECOVERY - Apache HTTP on mw1127 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 1.754 second response time [03:43:53] RECOVERY - Apache HTTP on mw1132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 2.870 second response time [03:43:54] RECOVERY - Apache HTTP on mw1196 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.037 second response time [03:43:54] RECOVERY - Apache HTTP on mw1206 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.042 second response time [03:43:55] RECOVERY - Apache HTTP on mw1133 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.232 second response time [03:43:55] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.337 second response time [03:43:56] RECOVERY - Apache HTTP on mw1137 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 6.235 second response time [03:43:56] RECOVERY - Apache HTTP on mw1123 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.161 second response time [03:43:57] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.212 second response time [03:44:02] RECOVERY - Apache HTTP on mw1147 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 7.091 second response time [03:44:02] RECOVERY - Apache HTTP on mw1115 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.188 second response time [03:44:02] RECOVERY - Apache HTTP on mw1143 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.236 second response time [03:44:22] RECOVERY - Apache HTTP on mw1140 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.189 second response time [03:44:22] RECOVERY - Apache HTTP on mw1120 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.190 second response time [03:44:22] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2944 bytes in 0.074 second response time [03:44:24] RECOVERY - Apache HTTP on mw1129 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.234 second response time [03:44:25] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.201 second response time [03:44:32] RECOVERY - Apache HTTP on mw1122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.165 second response time [03:44:32] RECOVERY - Apache HTTP on mw1139 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.213 second response time [04:11:21] yuck [04:45:31] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [04:55:40] (03CR) 10Legoktm: [C: 04-1] "Discussion says account blocks will be indef, and anon blocks are leaning towards 3 months. Current patch will make everything be 3 months" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98002 (owner: 10John F. Lewis) [05:20:52] (03CR) 10Legoktm: [C: 031] enable Echo on all beta.wmflabs.org-wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95450 (owner: 10Umherirrender) [05:21:49] * p858snake|l echo's legoktm out to the shell [05:43:18] wtf [05:47:15] paravoid: it was all legoktm's fault >.> <.< [05:47:25] hmm? [05:47:26] whaaaaa [05:54:28] !log rebooting cp1052 with the same kernel (control) [05:54:43] Logged the message, Master [05:55:10] PROBLEM - Host cp1052 is DOWN: PING CRITICAL - Packet loss = 100% [05:56:40] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [05:57:40] RECOVERY - Host cp1052 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [06:27:27] PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:29:27] RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active [07:10:37] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [08:00:27] ori-l: https://github.com/vimeo/graph-explorer [08:21:48] jeremyb: looks cool. want to puppetize? [12:30:43] (03CR) 10Faidon Liambotis: [C: 032] Setting up varnishkafka on mobile varnish caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/94169 (owner: 10Ottomata) [13:27:05] (03PS4) 10John F. Lewis: Enable AbuseFilter block option on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98002 [14:09:46] (03CR) 10Odder: Enable AbuseFilter block option on Wikidata (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98002 (owner: 10John F. Lewis) [14:27:39] PROBLEM - HTTP on fenari is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:26] RECOVERY - HTTP on fenari is OK: HTTP OK: HTTP/1.1 200 OK - 4775 bytes in 0.071 second response time [15:05:31] (03CR) 10Ottomata: Initial deb packaging (0313 comments) [operations/debs/python-kafka] (debian) - 10https://gerrit.wikimedia.org/r/97848 (owner: 10Ottomata) [15:06:03] hey [15:06:05] (03PS7) 10Ottomata: Initial Debian packaging [operations/debs/python-kafka] (debian) - 10https://gerrit.wikimedia.org/r/97848 [15:06:11] hey! [15:06:36] saturday morning work time woooo [15:06:53] hows it goin faidon?! [15:06:56] okay [15:07:15] how's the sleep vs. emergency paging these days? [15:07:33] less emergencies lately [15:07:46] but it is saturday and I've worked for > 8h now [15:07:53] tsk tsk tsk [15:07:54] so... :) [15:08:06] thanks for reviewing my stuff, much appreciated [15:08:13] no worries [15:08:17] will get new patchsets in shortly, but don't worry about getting to them today [15:09:50] (03CR) 10Faidon Liambotis: [C: 032] "(No idea about the (WMF), maybe Debian tools ignore it?)" [operations/debs/python-kafka] (debian) - 10https://gerrit.wikimedia.org/r/97848 (owner: 10Ottomata) [15:10:04] https://graphite.wikimedia.org/render/?title=HTTP%205xx%20Responses%20-8hours&from=-8hours&width=1024&height=500&until=now&areaMode=none&hideLegend=false&lineWidth=2&lineMode=connected&target=color%28cactiStyle%28alias%28reqstats.500,%22500%20resp/min%22%29%29,%22red%22%29&target=color%28cactiStyle%28alias%28reqstats.5xx,%225xx%20resp/min%22%29%29,%22blue%22%29 [15:10:09] finally [15:10:32] is that a normal level? [15:10:44] the blue one [15:10:44] yes [15:11:08] the red one, it's also normal but it's basically mediawiki [15:11:29] i haven't been following the 5xx saga there, was most of that packet loss issues? [15:11:34] nope [15:12:05] some code deployment? [15:14:25] it was packet loss, the strange GET \\ requests, the centralautologin stuff, ULS, the page cache kernel bug, mobile varnish crashing every 10', XFS kernel bug deadlock, varnish bug resulting in XFS fragmentation [15:14:32] that's the last two weeks [15:15:00] heh [15:15:22] well, plus other unrelated outages that cascaded, e.g. some database overloads, parser cache crash, elasticsearch issues, vectorbeta deployment [15:15:41] (also the last two weeks) [15:16:00] let's just hope december will be a better month [15:17:46] yeah [15:17:47] three varnish bugs, two kernel bugs, 4 network outages by the same vendor in two different sites [15:17:54] I mean, the odds... [15:17:56] ee yeah [15:29:43] paravoid: how about last nginx bugs? :) [15:29:51] which one? [15:30:06] http://mailman.nginx.org/pipermail/nginx-announce/2013/000125.html [15:30:25] (03PS3) 10Ottomata: Initial Debian packaging [operations/debs/logster] (debian) - 10https://gerrit.wikimedia.org/r/95556 [15:30:42] (03CR) 10Ottomata: Initial Debian packaging (039 comments) [operations/debs/logster] (debian) - 10https://gerrit.wikimedia.org/r/95556 (owner: 10Ottomata) [15:31:29] I don't think that affect us anywhere [15:31:38] nice [15:32:14] paravoid, got a +1 from alex on the varnishkafka .deb, once we are good with an actual tag, are you ok if I merge that? [15:32:42] ottomata: yes, but see magnus' latest commits [15:32:46] before you tag [15:32:56] ok cool, yeh i see them, som eneed merged still [15:33:03] (03CR) 10Faidon Liambotis: [C: 032] Initial Debian packaging [operations/debs/logster] (debian) - 10https://gerrit.wikimedia.org/r/95556 (owner: 10Ottomata) [15:33:05] we'll get that running for at least a day or something before we tag [15:33:54] VCL_Log:key is also useful [15:34:03] I want us to start using it [15:34:05] oh ok [15:34:06] cool [15:34:19] that can/should be done in puppet though, right? [15:34:23] not as a package default [15:34:54] btw, what is it? :) [15:34:59] that's varnishkafka support (r98135) + a VCL change + format string change [15:35:27] it's so that you can do from VCL std.log("analytics: zero=123") [15:35:34] without setting it as a response header (X-Analytics) [15:35:36] and logging that [15:35:38] ah coooool [15:35:50] so it just makes a varnishtag without mucking with http headers? [15:35:52] arbitrary logging from VCL basically [15:36:12] yes [15:36:13] (dunno if 'varnishtag' is correct term, but varnish stuff, not http headers) [15:36:19] cool [15:36:22] well, whatever you want to log [15:36:27] right aye [15:36:27] you can do all kinds of crazy shit in VCL anyway [15:36:40] but X-Analytics is one we can do [15:36:52] we set it now just so that we can log it [15:38:31] hm [15:38:33] question [15:38:45] on the varnishkafka change [15:38:46] format => [15:38:51] is space separated, not tab [15:38:52] is this right? [15:39:28] in the package, the default I set is tab [15:39:42] I'm talking about https://gerrit.wikimedia.org/r/#/c/94169/8/manifests/role/cache.pp [15:39:44] in puppetization change, its json, so neither. [15:39:51] ah, right [15:40:34] may I suggest a very trivial change? [15:40:36] sure [15:40:38] make it [15:40:42] format => join( [15:40:46] er, format => join([ [15:40:50] hmmmm, cool [15:40:51] i like that [15:40:51] ' 484 [15:40:55] yeah that will be much easier to read [15:41:11] "%{fake_tag0@hostname?${::fqdn}}", newline, ... [15:41:15] one per line [15:41:26] i coouuuuld make the puppet module fancier [15:41:35] and if it is passed a list, do that in the template [15:41:40] heh, I guess you could, yes [15:41:55] I don't mind either way, as long as we make it somewhat more readable :) [15:41:58] hm, would have to add a separator param if it was not json though [15:41:58] meh [15:42:30] so, remind me, if it's json does varnishkafka splits the string with spaces? [15:42:41] yes, possibly any whitespace, dunno [15:42:47] okay [15:43:16] and I thikn that varnishkafka will output the actual json data in the order that it is listed in the format string [15:43:37] even though that doesn't actually matter for parsing it, it is convenient to be able to read it the same way every time [15:43:45] nod [15:43:56] it's nice to see all of this coming along [15:43:59] finally [15:44:02] haha [15:44:07] uh huhhhhh FINALLY [15:44:09] haha [15:44:27] i was hoping to deploy a month ago, and then was going to deploy this monday and will probably deploy in a week, and then probably say that again in a motnh [15:44:30] hopefully soon! [15:44:34] but i'll believe it when I see it :p [15:44:36] you did see my comment about deploying gradually, right? :P [15:44:46] sounds familiar… [15:44:50] like, what, one mobile at a time? [15:44:52] I sent an email today [15:44:55] or at least, start with one [15:44:55] ah [15:45:00] haven't read all emails yet [15:45:09] ah reading [15:45:11] yeah, some form of that [15:45:21] like half of eqiad, then all of eqiad then esams [15:45:23] or something like that [15:45:25] but not all together [15:45:43] and wait a few days maybe between step 1 & 2 [15:46:25] yeah, good idea [15:47:10] oo on Kafak mailing list: [15:47:10] how's the consumer side coming along? [15:47:11] 'The 0.8 final release is being voted right now. If the vote passes, it [15:47:11] should be available next week.' [15:47:15] camus is it? [15:47:17] i haven't been working on it much [15:47:18] yeah [15:47:19] i mean [15:47:20] i have it working [15:47:22] it works great [15:47:23] but [15:47:27] its a hacky setup rigiht now [15:47:32] prebuilt .jar + user cron job [15:48:01] cron job? [15:48:04] oh, and also automatic hive partitioning [15:48:06] to consume? [15:48:08] yeah, camus is just a MR job [15:48:20] but, it doesn't matter when it is run, really [15:48:28] it'll launch a bunch of kafka consumer procs on the datanodes [15:48:32] each consuming from a topic-partition [15:48:51] and each will write the messages to proper time bucked hdfs dirs [15:48:57] based on content timestamp [15:49:05] then, dan and I wrote some other code (that has to be run separately) [15:49:23] to automatically look at the partitions of a hive table, and what dirs exist in hdfs [15:49:30] and then create whatever hive partitions need to be created [15:49:39] so, once that is all puppetized [15:49:54] it'll basically be automated queryable hive tables of webrequest logs [15:50:02] so how is linkedin doing it? [15:50:05] we'll probably run it once an hour [15:50:06] without yours/dan's code? [15:50:14] oh i doubt they are using hive much, I don't know [15:50:36] aha [15:50:46] so, wait, if we run it once an hour [15:50:53] you mean just leave logs being buffered inside kafka? [15:51:02] well, we can import as often as we want [15:51:13] but, the data is partitioned hourly [15:51:18] it doesn't actually matter [15:51:34] it just means that if we run more often, the latest partition won't be full once it shows up in hive [15:51:49] so if you query and do some kind of grouping on hour, it might look a little weird for the latest hour [15:51:52] buuut, it doesn't really matter [15:51:55] we can do however often we want [15:52:14] my camus cron job runs hourly right now, and it actually consumes from kafka pretty fast [15:52:19] but, that's just the 2 upload host's logs [15:52:22] so meh? [15:52:29] well, I'm thinking also some more realtime log analysis [15:52:36] sure, we can import as often as you like :) [15:52:45] well, almost [15:52:49] i think right now it doesn't matter [15:52:52] since we aren't using hadoop so much, [15:53:01] but, these are MR jobs, so they do take up some hadoop resources [15:53:27] if we run toooo often, the overhead time of launching the camus job could be greater than the time it actually takes to consume the data from kafka [15:53:32] heh [15:53:50] but we could probably do up to every 10 or 5 minutes or something [15:53:51] also [15:53:57] if you really want to to real time analysis [15:54:01] just consume directly from kafka [15:54:03] you can do that whenever you want [15:54:06] without affecting hadoop [15:54:15] I want both usually :) [15:54:20] yeah [15:54:29] well, I guess I can consume the whole kafka buffer [15:54:33] how long is it going to be? 7 days? [15:54:38] yeah [15:54:45] unless we change it, but that's the default, and it sounds good to me [15:54:52] we've def got room for that with mobiles for now [15:54:55] sure [15:55:05] wait until desktop comes though :) [15:55:08] heheh [15:55:08] yeah [15:55:24] it's about 10x I think [15:56:06] are there any alternatives to camus? [15:56:11] yeah i think that's right [15:56:28] sure, there's a hadoop consumer in contrib/ that ships with kafka (we aren't including it in our package) [15:56:31] but that is less useful [15:56:40] it just consumes from kafka into some directory [15:56:45] or we could write our own mr job [15:56:46] its not hard [15:56:52] camus is just nice because its a pluggable framework [15:57:01] i didn't have to write much to get it to work for our data [15:57:02] why is writing into some directory less useful? [15:57:05] just a coupel of short classes [15:57:16] no time buckets? [15:57:17] well, we'd like the data to be time bucketed in hdfs [15:57:18] yeah [15:57:20] right [15:57:55] so the time partitions in hive are mostly useful for optimziing your queries [15:58:04] if you know that you only want to look at the last month [15:58:09] you can tell hive that in your query in the where clause [15:58:24] and it knows to only read that data [15:58:27] rather than passing over all of it [15:59:10] oh, one other thing we'll have to do before we do regular analysis of these imports [15:59:14] is run a deduplication job of some kind [15:59:24] the imports as is could have duplicates in it, since kafka is at least one guaruntee [15:59:29] hopefully they will be rare [15:59:44] but, we will need to deduplicate before we use it for regular analysis [15:59:50] that's a todo on analytics side [16:00:06] (meaning, I don't really want to write it :p ) [16:00:40] heh [16:01:25] remind me, are we doing snappy per message? [16:01:29] or per 10 messages? [16:01:38] I remember such a discussion, but I forget the details now [16:01:47] its not per message, um, i think it is per batch size [16:02:07] that's internally to kafka, right? [16:02:19] hmmm to librdkafka/varnishkafka yes [16:02:23] there are two values [16:02:25] camus gets & stores decoded data, right? [16:02:28] a time and a batch size [16:02:31] er, decompressed [16:02:36] whichever it hits first [16:02:50] hmmm, no, data in hdfs is snappy compressed by default [16:02:52] so yeah [16:03:00] there are a lot of burnt cycles compressing and decompressing here [16:03:06] but we decompress & compress agian [16:03:07] right [16:03:28] varnishkafka compress, kafka broker decompresses when sending to camus, camus writes to hadoop, haddop compresses again [16:03:29] yeah [16:05:46] so, i think the batch size defaults are 1000 messages or 1000ms, whichever comes first [16:06:11] we could probably increase that, i think sending once a second is probably good [16:06:22] and the uploads are doing something like 2K messages per second, i thikn [16:06:56] more messages = better compression ratio [16:07:03] biiunno [16:07:19] it's all microoptimizations at this point I think [16:09:12]