[00:05:53] syddraf: i'm not an expert but to the best of my knowledge, the exit node has to be on the same ip as the website, which would require us to have exit nodes on all of our load balancers and that's a security risk (software wise) [00:06:14] if those requirements have changed, it would be worth looking into [00:10:14] LeslieCarr: I'm not an expert on these either, but I have been wondering why no websites are opening darknet nodes. I'll look more into it and see. Thank you. [00:11:02] New patchset: Asher; "ugly hack to get varnish purges working while w3/wp sends broken purge reqs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1941 [00:11:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1941 [00:15:16] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1941 [00:15:17] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1941 [00:19:19] gn8 folks [00:20:02] LeslieCarr: I've looked into it some more, and for Tor it looks like an exit node would be necessary unless the site is mirrored in Torspace. For I2P, if the I2P router is forwarded, from what I understand, your server system would treat it just like a normal access. [00:22:58] hrm, i2p could be worth looking into then , can you file a bug in https://bugzilla.wikimedia.org/ ? [00:23:24] mmmm wikipedia darknet :) [00:24:06] LeslieCarr: A friend of mine suggested for Tor to set up an exit node that does something similar, but our knowledge of Tor is much smaller than our knowledge of I2P. [00:26:19] LeslieCarr: Which product should I file it under? "Wikimedia"? [00:26:27] syddraf: most (if not all) TOR exit nodes are banned for editing [00:26:37] wikimedia sounds right syddraf [00:27:48] saper: I can understand that. That's an issue I'd already thought of and account-editing is the only solution I could come up with. [00:33:47] LeslieCarr: I've posted it with bug id 33779. Thank you very much. [00:36:25] New patchset: RobH; "added cp103X to mobile varnish range" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1943 [00:36:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1943 [00:40:16] New patchset: Lcarr; "tryuing to see where ganglia chokes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1944 [00:40:25] doh wrong pfr [00:40:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1944 [00:40:33] Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1944 [00:41:10] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [00:42:22] New patchset: RobH; "fixed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1946 [00:42:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1946 [00:44:58] New patchset: Lcarr; "another ganglia test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1947 [00:45:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1947 [00:45:22] Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1947 [00:47:07] New review: RobH; "once more with feeling" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1946 [00:51:09] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.043 second response time [00:58:18] Change abandoned: RobH; "redid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1943 [01:00:42] New patchset: Asher; "use mod_rpaf on blog server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1950 [01:00:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1950 [01:01:08] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1950 [01:01:09] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1950 [01:01:45] Change abandoned: RobH; "tired of dealing with it" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1946 [01:12:55] New patchset: Pyoungmeister; "this one is for comrad robh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1953 [01:13:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1953 [01:13:21] New review: RobH; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1953 [01:14:51] Change abandoned: Pyoungmeister; "wrong brach" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1953 [01:16:31] New patchset: Pyoungmeister; "for comrade robh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1954 [01:16:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1954 [01:16:57] New review: RobH; "once more with feeling!" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/1954 [01:17:26] New review: RobH; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1954 [01:17:26] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1954 [02:04:58] !log LocalisationUpdate completed (1.18) at Wed Jan 18 02:04:57 UTC 2012 [02:04:59] Logged the message, Master [02:12:00] New patchset: Lcarr; "fixing cp hosts so they will properly alert" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1955 [02:12:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1955 [02:13:47] Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1955 [02:15:48] New patchset: Lcarr; ""fixing cp hosts so they will properly alert"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1956 [02:16:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1956 [02:16:17] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1956 [02:16:17] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1956 [02:25:03] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1799s [02:29:03] New patchset: Asher; "this adds cp1039 and cp1040 to the varnish backend pool (add to pybal conf to also add frontend)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1957 [02:29:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1957 [02:29:33] PROBLEM - Memcached on marmontel is CRITICAL: Connection refused [02:30:05] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1957 [02:30:05] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1957 [02:34:53] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [03:28:52] !log laner synchronized wmf-config/InitialiseSettings.php 'Removing restriction of display title for SOPA landing pages' [03:28:55] Logged the message, Master [03:31:02] Aaron|home: you around? [03:31:35] I had one beer but I think I'm still OK [03:32:14] where is this fetching the data http://test.wikipedia.org/w/extensions-1.18/MobileFrontend/javascripts/application.js?version=01132011120915 [03:32:48] I mean this file: http://test.wikipedia.org/w/extensions-1.18/MobileFrontend/stylesheets/common.css [03:32:53] Aaron|home: ^^ [03:33:38] * Aaron|home looks [03:33:46] Aaron|home: because that is not the file in: /home/wikipedia/common/php/extensions/MobileFrontend/stylesheets/common.css [03:36:17] /home/wikipedia/common/php is a symlink to 1.17 [03:36:39] I wouldn't use the 'php' dir for anything if possible [03:36:59] nothing is running 1.17 either [03:38:35] Aaron|home: nope [03:38:39] Aaron|home: it's lrwxrwxrwx 1 tstarling wikidev 8 2011-10-15 19:37 php -> php-1.18 [03:39:02] Aaron|home: or, I mean php -> php-1.18 [03:40:04] hmm, right...I could have sworn I saw 1.17 when querying the symlink [03:40:44] Aaron|home: so, why does http://test.wikipedia.org/w/extensions-1.18/MobileFrontend/stylesheets/common.css not match that file? [03:43:55] vvv-away: i guess so, sure ;) [03:44:17] preilly: what changed in the file? [03:47:01] Aaron|home: this, ".mwm-message.mwm-notice {…" should be at the bottom [03:48:21] * Aaron|home doesn't see that in either file [03:49:11] Aaron|home: if you look on fenari in /home/wikipedia/common/php/extensions/MobileFrontend/stylesheets it is in common.css [03:49:23] just not at this link: http://test.wikipedia.org/w/extensions-1.18/MobileFrontend/stylesheets/common.css [03:50:37] Aaron|home: what does http://test.wikipedia.org/w/extensions-1.18 mount? [03:51:15] ok I was look in usr/ on fenari and didn't see it, but it's on /home [03:51:21] so you didn't sync anything yet I assume? [03:51:27] Aaron|home: correct [03:51:40] Aaron|home: I'm not ready to push it live [03:51:48] Aaron|home: is there a workaround for this? [03:53:20] s/look/looking [03:53:34] preilly: hmm, I don't know of one [03:55:33] Aaron|home: isn't that weird that it works like that? [03:56:19] testwiki is weird in a lot ways...CSS/JS, secure testwiki ;) [03:56:26] !log neilk synchronizing Wikimedia installation... : deploying CongressLookup (for i18n reasons, not deploying to enwiki) [03:56:27] Logged the message, Master [03:58:35] sync done. [03:59:06] neilk_deploying: dude. did you scap? [03:59:11] ah [03:59:12] had to [04:01:07] !log preilly synchronized php-1.18/extensions/MobileFrontend/ApplicationTemplate.php 'update version' [04:01:08] Logged the message, Master [04:16:03] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:24:53] RECOVERY - Disk space on es1004 is OK: DISK OK [04:33:41] !log neilk synchronized wmf-config/InitialiseSettings.php 'enable CongressLookup on enwiki' [04:33:42] Logged the message, Master [04:39:53] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [04:44:22] !log Disabling anon editing and page creation by users on enwiki for SOPA blackout [04:44:24] Logged the message, Master [04:45:10] !log laner synchronized wmf-config/InitialiseSettings.php [04:45:12] Logged the message, Master [04:47:18] !log Editing enwiki's MediaWiki:Robots.txt to disallow BannerController for SOPA blackout [04:47:20] Logged the message, Master [04:50:05] !log queuing up changes for totally disabling edits. DO NOT SCAP! DO NOT SYNC InitialiseSettings! [04:50:07] Logged the message, Master [04:51:47] Ryan_Lane: :O [04:58:00] T-2:00 and counting [04:58:56] :) [04:59:39] !log laner synchronized wmf-config/InitialiseSettings.php 'Disabling all editing for enwiki for SOPA blackout' [04:59:41] Logged the message, Master [04:59:48] Ryan_Lane: You're early! [05:00:04] Ryan_Lane: confirmed effective in the RC channel [05:00:43] Confirmed it works. [05:01:00] so, in the end, only enwiki did it. [05:01:13] ah, damnit [05:01:21] * DarkoNeko can still edit but know why [05:02:12] from -en: Moe_Epsilon I like how the blackout links you to [[Wikipedia:SOPA_initiative/Learn_more]] which is then blacked out [05:02:16] https://en.wikipedia.org/wiki/Wikipedia:SOPA_initiative/Learn_more needs to be whitelisted ... [05:03:08] it is [05:03:12] http://en.wikipedia.org/wiki/Wikipedia:SOPA_initiative/Learn_more isnt on the whitelist, can some one add it ? [05:03:32] uh, not [05:03:39] It's not black [05:03:45] reported in the right channel [05:04:02] very dark grey [05:04:23] Romaine: lol [05:04:37] :p [05:04:41] yay it's fixed :) [05:04:45] dark-greyout [05:04:57] I know that because I created this: http://nl.wikipedia.org/wiki/Wikipedia:SOPA [05:06:04] happy black-out-day [05:08:55] likewise [05:08:59] can the donation pages too be blacked out ? [05:09:40] ...why ? ^^; [05:09:43] http://twitter.com/ABVan/status/159501851493015553 [05:09:59] isnt it odd to accept donations on this day ? [05:11:13] ehm... Learn More is blacked out too :) [05:11:14] uh, that page is not even on enwiki [05:13:19] srikanthlogic: There was a thread on /. today about a bunch of people who wanted to donate during the black out to show their support appearently [05:13:26] why is there no banner on en-wiki: "if you pay, Wikipedia will be back" ? ;) [05:13:51] bawolff: ah.. [05:14:14] is it something leaking that I can send moodbar feedback now? en.wikipedia.org/wiki/Special:FeedbackDashboard/13938 [05:14:22] That's almost most definitly unrelated to why donations aren't blacked out, but thought i'd mention [05:14:22] http://en.wikipedia.org/wiki/Special:FeedbackDashboard/13938 [05:15:55] from my pov, the issue is different : the WMF isn't blackouting as a whole. Only one language, the neglish verison is. Cutting every wiki's donation page would be unfair [05:21:28] anyway, this is going to be a fun day [05:21:42] https://en.wikipedia.org/w/index.php?diff=470535638 If the Staff wants to add a Google+ link [05:21:48] * DarkoNeko provides cookies to the nice devs and techhies [05:21:55] s/Google+/Google plus one/ [05:25:00] !log laner synchronized wmf-config/InitialiseSettings.php 'Disabling moodbar on enwiki' [05:25:02] Logged the message, Master [05:27:27] Not a major issue given the whole sopa thing - but en.planet.wikimedia.org goes to apache's default "It works!" page [05:28:05] bawolff: can't reproduce [05:28:26] oh wait nevermind [05:28:49] sorry, that's embarrassing - its an issue on my end [05:28:54] I thought we fixed https planets... [05:28:56] no prob [05:32:40] !log laner synchronized wmf-config/InitialiseSettings.php 'Disabling CentralNotice for simplewiki' [05:32:41] Logged the message, Master [05:47:26] If someone here is capable of editing the English Wikipedia's blackout notice, please change "internet" to "Internet" and "zip code" to "ZIP code". Thanks! [05:48:06] #wikimedia-sopa [06:08:09] !log awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109319' [06:08:10] Logged the message, Master [06:13:53] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [06:24:39] !log awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109322' [06:24:40] Logged the message, Master [06:26:54] SpecialCongressLookup <- what. [06:28:03] !log awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109324' [06:28:05] Logged the message, Master [07:27:43] RECOVERY - Squid on brewster is OK: TCP OK - 0.000 second response time on port 8080 [07:45:57] !log awjrichards synchronized php/extensions/CongressLookup/SpecialCongressLookup.php 'r109336' [07:45:58] Logged the message, Master [08:01:52] PROBLEM - DPKG on db43 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [08:35:43] PROBLEM - Puppet freshness on db1045 is CRITICAL: Puppet has not run in the last 10 hours [08:43:13] looks like account creation on en is still up: http://en.wikipedia.org/wiki/Special:Log/newusers [08:43:49] for admins [08:43:52] well [08:44:02] i sose it's an admin-only thing anyway [08:44:05] pose [08:44:07] ugh [08:44:10] cannot type [08:49:16] !log neilk synchronizing Wikimedia installation... : [08:49:18] Logged the message, Master [08:52:39] sync done. [08:54:01] Is the English Wikipedia's mobile version supposed to be accessible (with a banner added)? Is the JavaScript-based content suppression not feasible in that format? [09:01:53] Guest1358_: yes it's supposed to be accessible [09:02:00] that was the plan [09:02:03] no editing of course [09:02:11] closedmouth: shouldn't be [09:03:31] Okay, thanks for clarifying, apergos. Out of curiosity, why was it decided to leave the mobile version available? [09:04:09] it was requessted that there be *some* way to access material; doing an appropriate blackout on mobile is much tougher [09:04:19] the constraints are might tighter for one thing [09:04:32] so that seemed like the logical place to leave access [09:04:45] Ah, I see. Thanks again! [09:04:49] closedmouth: but if you make an account on another wiki then you get one on en too [09:05:33] sure [09:05:35] i was thinking of something else [09:05:56] i see all of 1 entry there since the blackout... no big deal [09:05:58] http://en.wikipedia.org/wiki/Wikipedia:Account_creator [09:49:31] bwahaha [09:49:32] http://theoatmeal.com/sopa [09:52:19] wow [09:52:21] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 452125 MB (3% inode=99%): [09:52:29] I'm not always a huge fan of them but that was pretty awesome [09:52:51] do it for the kittens [09:59:41] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 423807 MB (3% inode=99%): [10:39:23] matanya, could you check https://meta.wikimedia.org/wiki/Special:Contributions/Rekp ? [10:39:50] [[he:User:Zivya]] has created this sock on translatewiki.net and other wikis [10:40:05] yes, shes ok [10:40:25] why socks [10:40:39] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:40:49] I'll ask here [10:40:57] *-e [10:45:18] droped a line [12:48:13] PROBLEM - Puppet freshness on mw1096 is CRITICAL: Puppet has not run in the last 10 hours [12:48:24] Localisation and internationalisation bug triage in 15 minutes in #wikimedia-dev. See http://etherpad.wikimedia.org/BugTriage-i18n-2012-01 for details. [12:58:02] PROBLEM - Auth DNS on ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:10:21] RECOVERY - Auth DNS on ns0.wikimedia.org is OK: DNS OK: 6.668 seconds response time. www.wikipedia.org returns 208.80.152.201 [13:21:20] New patchset: Hashar; "gallium: allow postgre restart" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1958 [13:43:47] !log demon synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php 'r109362' [13:43:48] Logged the message, Master [14:02:39] <^demon> Asav: itwiki has voluntarily blacked itself out. [14:03:20] <^demon> But the enwiki blackout shouldn't affect other language editions unless they decided to do something locally (with site stylings or banners) [14:03:52] Yes, but I'm trying to find out what impact the blackout has on other editions in terms of hits. [14:04:28] we don't really have graphs of that [14:04:33] the analytics team might [14:05:12] <^demon> Yeah, I was thinking the same thing. We don't track referrers via interwiki links. [14:05:47] <^demon> Which is how I could see the blackout affecting other language editions, and sounds like it prompted his question. [14:08:58] like, someone going to the local version instead of the usual English one? [14:09:24] <^demon> Or someone looking up something, going to enwiki, then following the language link to their local language wp. [14:09:26] interesting, that pageviews to enwiki aren't down much [14:09:44] <^demon> pageviews will still show since the blackout's being done with CentralNotice. [14:09:50] someone forgot to turn off our chinese laod generating bots [14:09:59] oh thanks for explaining ^demon [14:10:35] ^demon: is there a language link? [14:11:03] !log reedy synchronized wmf-config/InitialiseSettings.php 'Add Wikipedia বার্তা alias for aswiki' [14:11:05] <^demon> saper: I mean under normal circumstances if there's an interwiki link for other languages. [14:11:50] ^demon: good question if many people do so, I normally do the reverse (pl->{en,de} if the article is not satisfactory) [14:12:56] the only problem with a blackout today for me is that I could not show http://xkcd.com/926 to comeone [14:14:35] saper: use https … [14:14:57] (though is it ok to do that? …) [14:15:53] AVRS: thanks! Now I know I really meant https://xkcd.com/936 [14:16:44] saper: HTTPS Everywhere made me wonder if the blackout is JS-based [14:17:11] I am using duckduckgo for search which normally gives https links [14:22:25] request from last night from Sue -- what's the magic word or whatever that hides the breadcrumbs feature on a subpage? [14:23:02] <^demon> Is there a magic word to disable that? [14:23:23] I thought there was but it's a very vague memory [14:23:49] can't find it on mw.org but that doesn't mean it isn't in the code [14:26:26] some people complain since the morning about (pl)wiki being slow [14:26:50] saper: what wiki isn't slow? [14:27:11] (well, I am using https …) [14:27:33] e.g. Commons is slow for me [14:32:06] AVRS: dumps.wikimedia.org is still fast [14:32:50] lol [14:35:17] :D [14:35:28] New patchset: Mark Bergsma; "Disable holding all mail on sodium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1959 [14:35:30] Reedy: i downloaded a file with 80mbit/sec. so i can download all dewiki dumps in the same time i load one dewiki page [14:36:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1959 [14:36:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1959 [14:46:33] I wonder if the non-English language projects are busier today because of the en.wp blackout... [14:49:11] siebrand: did you see the internal thread? [14:49:29] where thread is just 1 msg [14:49:42] err, nvm [14:49:46] it was otrs-en-l [14:49:59] Asav: ^^^ [14:51:11] I figured it out. The page with the hourly visit dumps shhow that the Norwegian edition has approximately the same number of hits. Thanks all! [14:51:59] siebrand: ^ [14:53:55] jeremyb: I'm not on otrs-en-l [14:54:22] siebrand: i know (or guessed), but the person that sent it is Asav. ^^ [14:55:04] right ;) [15:04:18] Question. Listing global users like this works: http://commons.wikimedia.org/w/index.php?title=Special%3AGlobalUsers&username=B%20jonas [15:04:30] but? [15:04:36] but the form on the top of that page doesn't work: it actually takes you to the per-project user list [15:04:44] (to Special:ListUsers) [15:04:56] instead of to the same page (Special:GlobalUsers) [15:05:00] is that a known problem? [15:05:17] and if not, should I try to report it on mediawiki.org? [15:05:23] it's been reported at least 4 times [15:06:22] !b 31638 [15:06:23] https://bugzilla.wikimedia.org/show_bug.cgi?id=31638 [15:06:31] thanks [15:07:09] though that talks only about the pagination links, not the top form [15:08:26] but thanks for the link [15:08:34] this at least proves it's not just something on my side [15:15:03] New patchset: Mark Bergsma; "Google and possibly others are rate limiting our new ip, so use the old server(s) for delayed messages (for now)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1960 [15:15:33] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1960 [15:15:34] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1960 [15:17:38] PROBLEM - Host srv278 is DOWN: CRITICAL - Plugin timed out after 15 seconds [15:22:37] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [15:28:57] siebrand, dunno, but mattbuck says Commons is very slow for him, not even loading [15:32:05] front page of commons took a minute but after that other pages loaded pretty fast [15:32:26] hope it's not due to the banner! [15:32:37] loading the banner once you mean? [15:36:08] apergos, dunno. [15:36:13] apergos, Commons loads just fine for me [15:36:28] maybe it was a hiccup (one of my favorite explanations) [15:37:11] ;-0 [15:48:02] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [15:59:54] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [16:11:38] PROBLEM - Recursive DNS on 208.80.152.131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [16:22:27] RECOVERY - Recursive DNS on 208.80.152.131 is OK: DNS OK: 6.189 seconds response time. www.wikipedia.org returns 208.80.152.201 [16:23:45] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [16:57:47] !log reedy synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php 'r109395' [16:57:49] Logged the message, Master [17:25:07] RECOVERY - Host dataset1 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [17:26:14] hey [17:27:20] what's going on with the servers? massive slowness for deWP over both DFN (german university network) and commercial providers here and at some friend's? [17:29:16] same problem for dozens French users, since yesternday evening... [17:31:03] Watchmouse suggests commons for logged in, and bits is slow [17:33:12] New patchset: Lcarr; "adding in curl package" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1961 [17:33:25] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/1961 [17:34:07] Italian users report slowness across all wmf website more or less. [17:34:11] Me among them [17:34:16] Extreme slowness. [17:37:04] New patchset: Lcarr; "adding in curl package" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1961 [17:38:13] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1961 [17:38:14] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1961 [17:45:04] hmm lemme see how it looks from here [17:45:19] give me a sample url Snowolf? [17:52:55] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Wed Jan 18 17:52:36 UTC 2012 [17:58:39] New patchset: Asher; "fix vg naming on db builds, include new server range" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1962 [18:07:44] apergos: everything even simply http://en.wikipedia.org/wiki/Main_Page [18:07:51] or https://it.wikipedia.org/wiki/Template:Monitoraggio [18:08:22] PROBLEM - Recursive DNS on 91.198.174.6 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:09:30] hmm because for me here (I am in Greece) they load very quickly [18:09:39] Italy here / Telecom Italia [18:09:45] I have otenet [18:09:51] apergos: I'm getting a 40% packet loss over telia, everything ok via direct peering [18:10:01] I thought we did something about telia [18:10:10] can you give a traceroute please [18:10:21] and actually mention this in the #wikimedia-operations channel [18:10:25] leslie should look at it [18:10:29] anaconda: [18:10:57] hrm [18:11:04] that's "awesome" [18:11:07] or you could look at it here [18:11:16] :-) [18:11:19] traceroute + your ip, preferably in a pastebin so it doesn't take up 50 lines [18:11:36] apergos: i have an alert set for "leslie " :) how i knew [18:11:41] I figured [18:11:49] :-) [18:15:32] apergos: http://pastebin.com/00CNTVWd 87.5.17.85 [18:18:54] RECOVERY - Recursive DNS on 91.198.174.6 is OK: DNS OK: 9.285 seconds response time. www.wikipedia.org returns 91.198.174.225 [18:19:47] quick ? - i guess we're sending google + traffic incorrectly, can i pass on anyone's name to their team ? [18:20:19] if no one wants to jump in the wm sopa channel they can talk to me I guess [18:20:26] but I'll just relay the info basically [18:28:36] New patchset: Asher; "fix vg naming on db builds, include new server range" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1962 [18:28:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1962 [18:29:11] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1962 [18:29:12] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1962 [18:32:49] PROBLEM - Host dataset1 is DOWN: CRITICAL - Host Unreachable (208.80.152.166) [18:42:33] New patchset: Bhartshorne; "deploying a new SOPA filter and sending results to a new log file for Faulkner" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1963 [18:43:44] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1963 [18:43:45] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1963 [18:45:46] PROBLEM - Puppet freshness on db1045 is CRITICAL: Puppet has not run in the last 10 hours [18:51:41] zzz =__= [18:54:38] RECOVERY - RAID on db1004 is OK: OK: State is Optimal, checked 2 logical device(s) [19:16:08] New patchset: Asher; "preparing to upgrade two enwiki db's" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1964 [19:16:59] taking advantage that the load is low? [19:17:06] :) [19:17:09] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1964 [19:17:19] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1964 [19:24:16] New patchset: Asher; "fix regex" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1965 [19:25:29] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1965 [19:25:29] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1965 [19:26:16] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [19:28:04] !log Run patch-jobs-add-timestamp.sql on enwiki (jobs table is empty!) [19:28:05] Logged the message, Master [19:42:17] i need to dump a table from enwiki to a csv - how would i go about doing that? [19:42:42] Reedy, do you know? ^ [19:43:00] how big a table? [19:43:13] not huge , i think < 10000 rows [19:43:23] do you have access to the db where it runs? [19:43:30] if not you are going to have to convert it [19:43:45] it's a table on enwiki [19:43:56] enwiki.cl_errors [19:44:03] there's a mysql stmt that you can run if you have access on the db itself (on the host, write access into the dir where it puts things0 [19:44:13] ah 15787 rows [19:44:25] i dont have write access on the host [19:44:28] you'll need to convert it usin a script of some kind [19:44:29] then [19:44:57] yeah, i know i could use mysqldump to do it nicely, but not from a remote host [19:47:22] I have a crappy conversion script that might break but you could try it :-D [19:47:25] lemme see where that is [19:47:40] apergos [19:47:43] think ijust figured it out :) [19:47:46] oh? [19:47:48] let's hear it [19:48:29] sql enwiki -e "select * from cl_errors;" | sed 's/\t/","C/g;s/^/"/;s/$/"/;s/\n//g' > cl_errors-20120118-1948.csv [19:48:39] oh, if you have regular data [19:48:42] on fenari [19:49:42] yes, any host where you can save the output is good enough [19:51:44] :D [20:06:08] New patchset: Asher; "puppet is being tricky about overlapping node defs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1966 [20:06:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1966 [20:06:37] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1966 [20:06:38] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1966 [20:11:01] !log asher synchronized wmf-config/db.php 'pulling db38 from s1 for upgrade' [20:11:02] Logged the message, Master [20:26:08] !log asher synchronized wmf-config/db.php 'returning db38 to service' [20:26:10] Logged the message, Master [20:28:14] !log asher synchronized wmf-config/db.php 'pulling db38 again' [20:28:15]