[00:00:23] New patchset: Asher; "add mysql::packages to a db40-43" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1697 [00:00:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1697 [00:01:00] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1697 [00:01:00] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1697 [00:01:16] New patchset: Lcarr; "adding in UDP iptables service" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1696 [00:01:23] updated :) look better now ? [00:01:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1696 [00:03:48] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/1696 [00:03:52] lgtm! [00:04:05] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1696 [00:04:06] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1696 [00:06:20] maplebed: how goes swift optimizing? [00:06:38] I'm trying moving the container storage to ramdisk [00:06:42] interesting, it's not showing any rules... [00:06:43] to see if that's our bottleneck. [00:07:38] sqlite only had table-level locking, which probably doesn't help [00:07:41] *has [00:09:22] LeslieCarr: try running it twice? [00:09:34] ah, host/network all not found [00:10:17] interesting [00:11:42] so my rules appear to mimic the swift rules [00:12:11] oh i have a source rule of all [00:12:36] New patchset: Lcarr; "Changing source => all to blank" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1698 [00:12:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1698 [00:13:01] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1698 [00:13:01] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1698 [00:15:35] hrm, this is the line that fails -A INPUT -m comment --comment udp2log_drop_udp_udp -p udp -j DROP -s all [00:16:43] i guess i'll try specifying the source [00:16:59] New patchset: Lcarr; "specifying source of 0/0" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1699 [00:17:31] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1699 [00:17:31] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1699 [00:19:09] D: [00:19:43] that did it [00:19:44] yay [00:19:48] source 0/0? :P [00:19:49] locke is safe [00:19:51] yep [00:20:44] we did have a minor security hole, now plugged :) [00:31:44] since the spice (aka log traffic) is still flowing, time for me to yell at clipper [00:32:07] (you changed your credit card with autorefill and we didn't update it in time for the refill a month ago, now your card is shut down) [00:32:07] grrr [00:59:16] * aude wonders if there a reason wikipedia.org and the other domains are registered w/ godaddy? [01:01:10] http://judiciary.house.gov/issues/Rouge%20Websites/SOPA%20Supporters.pdf [01:25:44] New patchset: Asher; "move varnishncsa monitoring to cache::mobile, combine mobile cache node defs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1700 [01:26:22] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1700 [01:27:05] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1700 [01:51:08] New patchset: Asher; "write pidfiles and fix status option" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1701 [01:51:27] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1701 [01:51:27] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1701 [01:52:06] aude: who else should they be registered with? [01:53:16] TimStarling: would you mind poking at https://bugzilla.wikimedia.org/33323 ?? Ignore all my comments but the last one. [01:53:25] umm... for wikimediadc, we're choosing namecheap but there are other choices [01:53:52] sure [01:53:54] they are well regarded and don't support sopa [01:54:05] http://community.namecheap.com/blog/2011/12/22/we-say-no-to-sopa/ [01:54:07] godaddy supports sopa? [01:54:18] judiciary.house.gov/issues/Rouge Websites/SOPA Supporters.pdf [01:54:35] http://www.wired.com/threatlevel/2011/12/godaddy-sopa/ [01:54:59] it's kind of embarrasing that we oppose sopa, but then use godaddy [01:55:20] so someone upgraded rsvg? that always causes problems [01:55:37] TimStarling: outside of the lucid upgrade? [01:55:52] that's not clear [01:55:57] Ryan_Lane: I think it was the lucid upgrade [01:56:04] but yeah, not clear [01:56:06] probably was. [01:56:20] it's like upgrading tidy [01:57:00] not saying it's a bad idea to upgrade, obviously it has to be done sooner or later [01:57:09] aude: and their CEO also hunts elephants. But I used 'em even after that. It is interesting that GoDaddy is the only tech company on there. [01:57:29] yeah [01:57:45] well, I kind of doubt that it happened outside of lucid upgrade, unless there was a security reason for doing so [01:58:50] definitely not a good idea to close any rsvg bugs as "invalid" [01:58:56] we're basically comaintainers for rsvg [01:59:05] heh [01:59:09] ah, k. good to know [02:00:24] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 952s [02:01:26] anyway I'm not sure what you want from me on this issue [02:01:33] do you want me to isolate it? [02:02:53] if it's fixed in the latest rsvg we can just backport it to lucid [02:04:14] TimStarling: I'm not sure *what* to do. I'm not sure what the ealier version we had was but earlier versions don't show the problem. Nor does the current one in Ubuntu [02:04:23] guess I need to check lucid now [02:04:32] * hexmode goes off to check [02:05:18] isolating it properly would mean that we could just backport that patch [02:05:35] or you could do some version voodoo and we could backport the whole new upstream version [02:06:28] let me check the lucid version [02:06:53] the image scalers were the first to upgrade weren't they? [02:06:54] Ryan_Lane: did you guys do any custom patches on rsvg? [02:07:04] I very much doubt it [02:07:25] unless we were asked to [02:07:35] hell, it's probably whatever comes with lucid [02:08:04] hmm, that would be quite bad [02:08:04] 'swhat I thought [02:08:09] we have security patches [02:08:35] of course, I'm just guessing here. I didn't do most of this work. [02:08:55] Ryan_Lane: do you know offhand who did? [02:09:02] notpeter did [02:09:08] mark did some, I did some [02:10:11] TimStarling: are the patches available somewhere? [02:10:13] you know, you can look on a srv system to see this [02:10:16] of course [02:11:22] Ryan_Lane: srv? you don't mean labs do you? [02:11:26] the patches werent upstreamed? [02:11:42] there isn't a single instance that starts with srv... [02:12:16] I mean, you can take a look at an apache that's in production [02:15:44] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1872s [02:25:34] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:29:24] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:53:04] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Fri Dec 23 02:52:44 UTC 2011 [03:26:53] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Fri Dec 23 03:26:37 UTC 2011 [04:46:56] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [09:02:41] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [09:15:21] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [09:57:21] RECOVERY - MySQL slave status on es1004 is OK: OK: [13:01:52] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [13:23:32] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 5 processes with args varnishncsa [13:41:47] RECOVERY - mobile traffic loggers on cp1042 is OK: PROCS OK: 4 processes with args varnishncsa [15:19:07] bah, I had hoped the find was taking the dirs in commons/thumb in order, but no, of course not [15:19:13] so it's only about half done :-/ [15:28:49] hexmode: what could you possibly have done to google? They hate you. [15:45:53] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Fri Dec 23 15:45:33 UTC 2011 [15:57:39] waiting for gerrit.wikimedia.org ..hmmm [16:10:30] !log gerrit stopped working and formey would still ping but no ssh connect and no mgmt output, powercycling formey [16:10:40] Logged the message, Master [16:11:23] PROBLEM - HTTP on formey is CRITICAL: Connection refused [16:13:09] !log gerrit and svn back up [16:13:17] Logged the message, Master [16:21:03] RECOVERY - HTTP on formey is OK: HTTP OK HTTP/1.1 200 OK - 3596 bytes in 0.008 seconds [16:24:55] woosters: hi, we just had a short outage of gerrit and svn, formey hung, rebooted it and back [16:25:13] yes, got that page [16:25:36] any telltale signs why? [16:25:44] all seems to be working again, couldnt do much besides powercycle cause their was no mgmt output when it was gone [16:25:58] there [16:26:05] still checking ..now [16:27:16] woosters: linux kernel panic [16:27:52] no, not really panic, but "Dec 23 15:53:46 formey kernel: [39665413.570024] INFO: task kswapd0:36 blocked for more than 120 seconds." [16:29:47] looks like there was a spike in load before it rebooted [16:30:18] !log first interesting syslog line when it started: formey kernel: [39665413.570024] INFO: task kswapd0:36 blocked for more than 120 seconds. [16:30:28] Logged the message, Master [16:30:35] yea [16:37:45] !log kswapd crashed, there is a Call Trace, and there was a load spike before, guess it is https://bugs.launchpad.net/ubuntu/+bug/721896 or similar [16:37:53] Logged the message, Master [16:39:24] woosters: most likely an Ubuntu bug, that confirmed one is pretty similar, guess that's all for now [16:40:11] ok [18:14:53] Brownout: you mean my email in spam? [18:15:03] * hexmode just got back [18:16:48] mutante: was it you that did the rsvg upgrade in ubuntu? [18:21:02] hexmode: yes, every time I log into the webmail there are a bunch of your messages in the spam folder [18:21:35] Brownout: I hope you tell the big G I'm not spamming you :) [18:21:52] but, yes, evidently that happens to a lot of my email [18:22:32] I do, maybe I should add your address in my contacts or create a filter [18:25:22] Brownout: if you find out what keeps 'em out of spam, let me know so I can tell others when they say something to me [18:26:07] just have people mark as not spam + if you use gmail there's the new stupid circles feature you can put someone into [18:26:41] I'm not on g+ [18:26:57] circles! [18:26:58] i think it's on gmail itself, not google + [18:27:02] new feature [18:27:12] oh wait no [18:27:18] it's adding your google + circles into gmail [18:36:41] hexmode: just following up about the "need to reindex to have gender handled correctly" bug [18:37:01] is it closed? also, can you relink me to the bug? [18:37:30] notpeter: 1s ... was looking at another operations issue [18:37:38] * hexmode goes to pull up the bug [18:37:44] ok, cool [18:37:46] no hurry [18:38:13] http://bugzilla.wikimedia.org/31697 [18:38:27] notpeter: ^^ [18:39:29] notpeter: was it you who upgraded rsvg? [18:41:36] hexmode: is there another related bz ticket? I can set things up for reindexing, but as rainman pointed out, we need to know how far back the reindexing needs to go... [18:41:41] hexmode: uh... it's possible? [18:41:56] I did upgrade the imagescalers, I believe [18:42:03] oo! [18:42:12] so that would have involved some amount of upgrading rsvg, I guess [18:42:30] the version of librsvg2-bin (which has rsvg) in Lucid has a bug [18:42:39] ah, ok [18:42:45] I'm looking for upstream fixes [18:42:51] anyway, reindexing [18:43:23] notpeter: what do you mean "how far back the reindexing needs to go"? [18:43:28] hexmode: ok, keep me posted about needing to rebuild the librsvg2 package [18:43:47] If you reindex, do you not need to reindex the whole corpus [18:43:59] * hexmode hopes he used "corpus" right [18:44:55] are there any other bz tix related to http:// rt.wikimedia.org/Ticket/Display.html?id=2160 [18:49:18] notpeter: I don't think there are any other tickets, no [18:51:00] hexmode: maybe my dreams are becoming to real and mundane.... "last night, I had a dream about the most average bugzilla ticket..." [18:51:09] oh! I have the scrollback I need, I think [18:51:12] one sec [18:51:55] heh [18:58:06] notpeter: could you do dpkg -l librsvg2-bin on an imagescaler and tell me if it is version 2.26.2-0ubuntu1 [18:58:08] ? [18:59:17] hexmode: 2.26.3-0ubuntu1.1 [18:59:33] same bug [18:59:40] boo [18:59:52] at least 2.26.3 on debian has the bug [19:00:19] ubuntu often backports sec patches [19:01:54] notpeter, hexmode: there's an open RT on this [19:02:03] I opened it last night [19:02:11] Ryan_Lane: rsvg? [19:02:12] let's try not to talk about that RT in this channel ;) [19:02:48] Ryan_Lane: rt #? [19:02:54] 2190 [19:02:59] please join the other channel [19:03:15] Ryan_Lane: you mean, the one I can't join? [19:03:25] well, then get a cloak, and get added to that channel [19:03:34] there's no reason you shouldn't be able to [19:03:57] I used to be, then something (I've no clue what) happened [19:04:18] hmm you have a cloak all right [19:05:16] apergos: Ryan_Lane: can one of you invite me in? [19:05:35] I don't know how to use IRC ;) [19:06:08] heh [19:06:35] I am never ops in thes things but I'm trying to figure it out [19:07:14] Ryan_Lane: so, without getting into specifics, if we're going to recompile, Just need to make sure we get the newer rsvg [19:07:32] well, put what you want into the ticket [19:07:39] will do [19:07:44] it's more than just recompiling, though [19:11:06] but if that is part of the process, then we can fix the other bug that Tim and I talked about last night :) [19:11:51] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [19:13:18] hexmode: ... anything? [19:18:47] Ryan_Lane: who's the volunteer who's working on search? [19:18:56] oren [19:19:00] nick? [19:19:09] lemme see if I can find this info [19:19:09] slash email address? [19:19:19] slash any way to contact this (I'm assuming) human? [19:24:51] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [19:29:27] hexmode: if you look at rt2160, and look at roan's comment, he points out that "some of the bad data is stuck in the indexer". to fix this I can reindex everything from just before that bad data got in there. but I'm not sure what date whatever change was made that got the bad data in there in the first place. do you know who I could ask to find that out? [19:30:38] notpeter: it would be whenever 1.18 was deployed. or shortly thereafter, since this bad data is related to gender, right? [19:31:06] notpeter: but otherwise I would ask Roan [19:31:09] that is the impression that I am under [19:31:27] kk, will do [21:05:30] notpeter: did you find oren? [21:06:13] he doesn't seem to have a USERINFO yet [21:06:29] that last few times i went searching for him i didn't find him on IRC [21:07:08] the last* [21:08:47] notpeter: i have summoned the oren :) [21:08:57] nice job! [21:09:06] hello good people [21:09:12] OrenBochman: hey, long time no see [21:09:24] indeed [21:10:11] upgrading code is much harder than writing new code [21:11:01] i was trying to figure out if amir's mail's '!' was applicable to both langs or just one :P [21:11:12] although idk if hebrew has a ! [21:11:23] anyway, chag sameach [21:12:57] heberew uses ! [21:13:06] thanks you too [21:13:44] jeremyb: are you involved with the coming wikimania? [21:14:40] OrenBochman: yes; there's others (some lurking here like aude) more involved but i am doing some stuff [21:15:13] good news [21:16:31] OrenBochman: http://dpaste.com/677223/ [21:37:37] Hello out there. I am going to be on call for the fundraiser on Christmas eve and I do not have access to #mediawiki_security. [21:37:57] Does anyone know how I can get access? [21:39:05] jpostlethwaite: yeah. gimme a sec. [21:39:13] thanks [21:39:22] jpostlethwaite: you need a cloak. [21:39:36] do i need a cloak when I am in the SF office? [21:39:51] nope. [21:40:00] i am in the office right now [21:40:01] presumably you won't be in the office on christmas eve? [21:40:06] correct [21:40:24] when i try to connect now, it says invite only [21:40:28] jpostlethwaite: join #wmfstaff; we'll pick this up here. [21:40:31] s/here/there/ [21:40:37] ok [22:22:14] OrenBochman: hey, you're the person who's working on revamping search?> [22:22:44] yes [22:23:43] are you Petrb? [22:23:44] sweet. I was going to at least puppetize the existing search conf. but if you're going to work on improving, I'd love to work with you [22:23:46] nope [22:23:51] pyoungmeister [22:24:04] ok [22:24:47] I have not been able to get search to work on my PC [22:26:04] however I am now upgrading the code from lucene 2.3.x to 2.9.4 [22:26:13] oh, awesome [22:26:22] this should fix servral bugs [22:26:35] I hope to get it working on windows as well [22:28:11] once I have an environment for testing I'll review and integrate the patches from bugzilla. [22:28:21] ok, cool. [22:28:35] it seems like we're working on different but complimentary things [22:28:42] good [22:28:54] I'm probably going to spin up a labs instance soon to test my puppet confs [22:29:21] so if we were working on the same labs instance that could be very good [22:31:06] I think that I will need some help with environments. [22:32:11] cool. we can get that going. [22:32:28] do you want to shoot me an email so we can communicate about this? [22:32:59] sure what is your email? [22:33:10] py@w.o [22:37:02] I'm just setting up a the project's contact list on my mediawiki page. [22:40:00] notpeter: I think that email is incomplete [22:40:37] er, yeah, sorry. @wikimedia.org [22:40:59] ok [22:41:14] lazy typing :) [22:42:26] notpeter: are you familier with wikitrust ? [22:43:01] nope [22:44:21] never mind It is not important right now [22:45:32] ok