[00:07:34] Hello, I have found an oddity with the possibility of being a secuirty issue [00:07:42] security* [00:07:51] I am not too sure as to where to report this [00:08:19] anybody here know how to report secuity bugs/issues to the wikimedia ? [00:10:11] can you /msg it to me? [00:11:17] DrGranit: Either pm it to werdna, or you can email security@wikimedia.org, or you can file it in bugzilla under the "Security" option [00:12:24] thanks [00:35:22] !log asher synchronized wmf-config/db.php 'returning db38 to s1 post schema migration testing' [00:35:24] Logged the message, Master [00:51:35] New patchset: Bhartshorne; "grrr... missing definition" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2421 [00:51:55] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2421 [00:51:55] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2421 [00:59:19] there we go [00:59:27] gzuufy_ - you had a question about encrypted logins or something? [01:00:33] There seems to be an issue with HTTPS no longer working. I'm just wondering if this is a problem on my side, or with wikipedia server software? [01:01:04] gzuufy_ - now you wait until someone notioces and answer. [01:01:28] which may take a while [01:01:41] gzuufy_: woudl you define 'no longer working'? https://en.wikipedia.org/wiki/Main_Page loads just fine for em. [01:01:45] s/em/me/ [01:02:34] Seems fine for me. I just re-logged in to https://en.wikipedia.org a few minutes ago. [01:03:12] Maplebed: Links are no longer automatically converted to https from, for example, "My watchlist". [01:03:55] gzuufy_: you mean that if you start at https://en.wikipedia.org/wiki/Main_Page and click on 'My watchlist', it loads as http, not https? [01:04:12] Maplebed: Yes. [01:04:45] huh. wfm. Have you tried a different browser? [01:05:38] I'm mistaken. You have helped me. Thank you. And yes, there's a browser upgrade in the middle of this issue. I'm sorry for any time wasted. [01:05:45] :) [01:05:47] np. [01:37:44] New patchset: Bhartshorne; "taking owa out of the prod swift cluster until we need them." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2422 [01:38:04] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2422 [01:38:04] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2422 [01:39:36] * Shirik blinks at the thought of a code review by the author [01:59:50] Shirik: though all checkins go through the code review process, not all of them are really worthy of multiple eyes. I know, the line is fuzzy, but so it goes. [02:01:21] it is nice, though, to be able to see the checkins from last $night (for various TZ definitions of night) and glance through them for interesting stuff. [02:01:30] even if they're not reviewed in realtime by others. [02:02:58] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 607s [02:26:50] !log LocalisationUpdate completed (1.18) at Thu Feb 9 02:26:49 UTC 2012 [02:26:53] Logged the message, Master [02:44:12] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 20s [03:32:32] RECOVERY - Puppet freshness on srv226 is OK: puppet ran at Thu Feb 9 03:32:22 UTC 2012 [05:21:03] hi [05:22:09] zzz [05:23:17] oh...good night [07:57:14] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [07:57:14] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [10:02:26] New patchset: Mark Bergsma; "Generate varnish node entries from a function" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2423 [10:02:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2423 [10:02:53] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2423 [10:02:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2423 [10:09:28] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2424 [10:09:49] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2424 [10:09:50] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2424 [10:09:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2424 [10:12:00] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2425 [10:12:19] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2425 [10:12:19] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2425 [10:12:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2425 [10:16:59] New patchset: Mark Bergsma; "Try Array.inspect" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2426 [10:17:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2426 [10:17:30] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2426 [10:17:31] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2426 [10:21:01] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2427 [10:21:20] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2427 [10:21:20] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2427 [10:21:21] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2427 [10:27:12] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2428 [10:27:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2428 [10:27:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2428 [10:27:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2428 [10:29:26] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2429 [10:29:46] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2429 [10:29:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2429 [10:29:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2429 [10:35:27] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2430 [10:35:47] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2430 [10:35:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2430 [10:35:53] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2430 [10:35:54] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2430 [10:37:49] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2431 [10:38:09] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2431 [10:38:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2431 [10:38:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2431 [10:39:16] New patchset: Mark Bergsma; "Formatting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2432 [10:39:36] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2432 [10:39:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2432 [11:01:36] New patchset: Mark Bergsma; "Generate all varnish nodes in varnish.xml from a function" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2433 [11:01:55] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2433 [11:03:38] New patchset: Mark Bergsma; "Generate all varnish nodes in varnish.xml from a function" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2433 [11:03:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2433 [11:04:07] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2433 [11:04:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2433 [11:08:37] New patchset: Mark Bergsma; "Corrections" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2434 [11:08:57] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2434 [11:09:05] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2434 [11:09:06] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2434 [11:21:12] New patchset: Mark Bergsma; "Make squid.xml generated by a template as well" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2435 [11:21:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2435 [11:21:51] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2435 [11:21:52] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2435 [11:29:11] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [11:34:01] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [11:36:01] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [11:56:01] RECOVERY - Host cp1017 is UP: PING OK - Packet loss = 0%, RTA = 26.42 ms [12:03:13] New patchset: Mark Bergsma; "Make sure Squid doesn't automatically start at boot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2436 [12:03:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2436 [12:03:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2436 [12:07:30] New patchset: Mark Bergsma; "Using the 'enable' service resource type parameter is a better way" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2437 [12:07:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2437 [12:07:57] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2437 [12:07:58] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2437 [12:30:28] New patchset: Mark Bergsma; "Automatically generate cachemgr.conf content from active_nodes Puppet list" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2438 [12:30:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2438 [12:30:58] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2438 [12:30:59] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2438 [12:35:35] New patchset: Mark Bergsma; "Sort lists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2439 [12:35:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2439 [12:36:05] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2439 [12:36:05] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2439 [12:49:22] !log around 11:05 UTC, increased /proc/sys/vm/min_free_kbytes from 16259 to 262144 to check impact on network alloc issues, on dataset1001 [12:49:24] Logged the message, Master [13:15:43] PROBLEM - Host cp1017 is DOWN: PING CRITICAL - Packet loss = 100% [13:43:43] does anyone know where are the files for http://www.mediawiki.org/xml/export-0.5/ ? :) [13:43:51] can't find them in puppet :/ [13:44:00] nor on fenari [13:44:42] I used to know, let me see [13:44:46] I think I might have put them there [13:47:24] ahh it is in the docroot :) [13:47:28] /usr/local/apache/common/docroot/mediawiki/xml/ [13:47:34] no I need to find out the source of that docroot [13:47:35] RoanKattouw: [13:47:59] /home/wikipedia/common/docroot/mediawiki/xml [13:48:21] You can sync with sync-common-file [13:48:26] I was in /home/wikipedia/htdocs … seems to be a bit old [13:49:46] thanks Roan [13:52:13] !log hashar synchronized docroot/mediawiki/xml [13:52:15] Logged the message, Master [14:02:59] PROBLEM - Auth DNS on ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:06:00] !log hashar synchronized docroot/mediawiki/xml/index.html 'Give a home page to our XML namespaces homepage' [14:06:01] Logged the message, Master [14:10:54] !log hashar synchronized docroot/mediawiki/xml 'update html pages for 0.1 and 0.2 schemes' [14:10:56] Logged the message, Master [14:14:45] New patchset: Mark Bergsma; "Update creator-info" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2440 [14:15:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2440 [14:15:09] RECOVERY - Auth DNS on ns1.wikimedia.org is OK: DNS OK: 0.028 seconds response time. www.wikipedia.org returns 208.80.152.201 [14:15:49] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2440 [14:15:49] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2440 [14:17:48] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34253 - Change site logo for lb.wiktionary' [14:17:50] Logged the message, Master [14:23:31] Reedy: thanks :) [14:24:27] np [14:27:36] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34183 - add alias redirection in ml wikisource' [14:27:38] Logged the message, Master [14:39:39] New patchset: Mark Bergsma; "Improve total requests graph" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2442 [14:40:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2442 [14:40:05] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2442 [14:40:05] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2442 [14:45:14] New patchset: Mark Bergsma; "Apparently AREA didn't work, despite torrus docs suggestion" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2443 [14:45:50] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2443 [14:45:50] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2443 [14:55:45] New patchset: Mark Bergsma; "Change eqiad colors for clarity" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2444 [14:56:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2444 [14:56:45] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2444 [14:56:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2444 [15:22:15] New patchset: Hashar; "rt: force HTTPS protocol" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2446 [15:22:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2446 [15:26:44] New review: Dzahn; "aah, i'd love to see that work, but please read through RT 714 for possible problems and why this ha..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2446 [15:33:21] New patchset: Mark Bergsma; "Add aggregate stats for mobile caches" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2447 [15:33:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2447 [15:34:50] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2447 [15:34:51] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2447 [15:44:09] mutante: so I guess I should abandon my change to make RT https only? :D [15:46:35] hashar: hmm, not so quick, we want it..but should check twice [16:45:23] PROBLEM - check_nginx on payments4 is CRITICAL: PROCS CRITICAL: 0 processes with command name nginx [16:50:23] PROBLEM - check_nginx on payments4 is CRITICAL: PROCS CRITICAL: 0 processes with command name nginx [16:55:23] RECOVERY - check_nginx on payments4 is OK: PROCS OK: 50 processes with command name nginx [16:57:41] New patchset: Andre Engels; "More on my own version of the pipelines (simplepipelines.py), new class MultiVariable, new file selectors.py with standard selectors of which log lines to include. apireturn.py, containing some code that I have used for analyzing the return MIME types of " [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2452 [16:59:55] Having trouble opening any Wikipedia or other Wiki sites. [17:01:41] Reported it even to bugzilla [17:01:47] No progress [17:02:03] It's just specifically for the Wiki websites. [17:02:14] Other websites are working fine. [17:02:51] Tried different browsers and computers. All show the same problem. [17:03:01] Any solutions? [17:03:11] https://bugzilla.wikimedia.org/show_bug.cgi?id=34256 [17:03:51] apparently it's "INVALID" and "RESOLVED" [17:04:30] How can it possibly be resolved? [17:04:48] I am facing the problem even now. [17:06:43] is it resolved? [17:06:49] New patchset: Mark Bergsma; "Change RRD RRA sizes for torrus usage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2453 [17:07:11] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2453 [17:07:21] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2453 [17:07:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2453 [17:07:36] No PiRSquared [17:09:07] the only problem I have is apparently with the API if I use the move&replace script on Commons... doesn't finish (but does the actions) [17:09:43] ah that.. the photobucket screenshots - I remember [17:10:12] Yes Saibo. The same ones [17:10:15] Snake_lover: did you ask your ISP? [17:10:27] Yes [17:10:45] I guess they said: if other websites are working it is not our problem ;) [17:10:52] They said everything is fine on there side. [17:11:04] hehe yes Saibo :D [17:11:09] of course [17:12:20] the problem is: apparently only you have that problem... don't you have a computer science friend? ;) [17:12:28] call him and let him look :D [17:13:08] I have. [17:13:19] He said start using a proxy [17:13:23] hmm [17:13:32] But I can't access my account from it [17:13:38] yes [17:13:50] Otherwise I will be accused of Sockpuppeting [17:14:06] and: proxy is slow [17:14:28] "same computer different connection" works, right? [17:14:42] just all devices with your connection do not work [17:14:47] Yeah [17:14:51] Yup [17:15:04] Specifically only for Wikipedia. [17:15:17] All others are working as they should be. [17:15:31] you said: "Bigger pages like Taj Mahal, Bill Gates, Steve Jobs etc are not even opening." [17:15:38] Even the ones supporting Flash [17:15:41] Yes [17:15:45] ah - you mean the articles! [17:15:48] :D [17:16:00] not the website of those persons [17:16:11] It takes more then 5 minutes to open up [17:16:29] Any page on Wikipedia, Wikimedia, Wiki everything [17:17:04] Just quoted the bigger pages that they don't even open after 5 minutes. [17:17:24] The browsing just stops and a blank white page appears. [17:17:26] http://bits.wikimedia.org/en.wikipedia.org/load.php what do you see on that page? [17:18:43] You would have to wait for 5 minutes to see the result ;) [17:19:11] ah.. [17:20:10] error [17:20:23] The requested URL could not be retrieved [17:20:23] New patchset: Andre Engels; "Typos in my previous commit." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2456 [17:20:23] just for trying it: did you try to use another DNS server? Probably not fixing it - but worth a try [17:21:06] I have tried everything that I could possibly think of. [17:21:37] Different browsers, computers everything [17:21:52] but no other DNS ;) [17:22:22] which operating system do you use? [17:22:32] Windows 7 [17:22:33] W7 [17:22:36] sorry ;D [17:22:42] saw it on the screeenshots [17:22:48] ;) [17:23:16] you can try to follow these steps: https://store.opendns.com/setup/operatingsystem/windows-7 [17:23:55] to get back to normal you need to check "automatically" again in the window shown in the last step [17:24:12] and probably a restart is best after changing [17:25:18] Oh, I did this thing. I didn't know the name. The technical guy on my ISP team told me to do it. [17:25:43] okay - and you entered those numbers? or the ones from your ISP? [17:25:59] The ones from ISP [17:26:09] then try those! ;) [17:26:15] Okay. [17:26:17] probably it will not help - it is just a try [17:26:30] I'll try :) [17:28:34] Snake_lover: just a security note: if you enter a malicious DNS server it can hijack all your internet connections - so don't enter numbers there a stranger (like me) tells you. ;) for verificaton: https://en.wikipedia.org/wiki/Opendns [17:29:52] I seriously wish I could open this Wikipedia page [17:30:01] ah.. em.. [17:30:05] let me think :D [17:30:10] Don't use openDNS, they pollute the DNS-system [17:30:22] damn it [17:30:23] Ok [17:30:23] :D [17:30:38] hi DaBPunkt - what do you mean? [17:30:49] Snake_lover: maybe that page works: http://en.m.wikipedia.org/wiki/Opendns [17:31:25] Saibo: they have a kind of "spelling correction system" if you mistype a domain-name (which breaks some DNS-functions) [17:31:28] DaBPunkt: also opendns was just for a test [17:31:30] /* No modules requested. Max made me put this here */ [17:31:31] ? [17:31:45] I have opened the link. [17:31:46] DaBPunkt: I know - but that is way better than the current situation for Snake_lover [17:31:56] Saibo: ok [17:32:15] DaBPunkt: that "spelling correction" is nasty.. I know [17:32:30] DaBPunkt: if you register at them you can switch it off, btw. [17:32:40] (still do not use them) [17:32:51] I use 8.8.8.8 as test-server (it's by google), but I also a stranger ;) [17:32:56] google.. lol [17:33:03] great joke [17:33:04] :D [17:33:26] they do not need to track any and every page view [17:33:33] just the advert pages are enough [17:33:48] Same DNS are here [17:33:50] 208.67.222.222 [17:33:59] and 208.67.220.220 [17:34:02] yes, that is fine [17:34:04] try it [17:34:13] I just did [17:34:13] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:23] Snake_lover: https://store.opendns.com/settings/premium-dns/ qhat does that page say? "Oops. You're not using OpenDNS yet."? [17:36:02] Yeah [17:36:50] then.. please just restart your pc - or enter that in a command window: ipconfig /flushdns [17:37:00] (cmd) [17:37:34] Success! You're now using OpenDNS. [17:37:39] okay, fine [17:37:43] now - try wikpedia [17:39:44] It's taking time [17:40:14] Nope a page opened but same result. [17:40:19] yup, okay [17:40:27] then you can change it back to "automatically" [17:40:33] the DNS server settings [17:40:37] Okay [17:40:59] Done [17:41:23] https://store.opendns.com/settings/premium-dns/ should tell you now the "Oops. You're not using OpenDNS yet." message again [17:42:37] well, get another ISP ;) [17:42:47] I wish. [17:43:27] Thanks Saibo [17:43:29] a good workaround is maybe indeed a good proxy [17:44:16] Snake_lover: a simple and fast proxy without sock problems (probably): use http://en.m.wikipedia.org/wiki/Opera_%28web_browser%29 in "Turbo" mode [17:44:37] this proxy is trusted by wikimedia - so you will not be blocked [17:44:57] Also on Wikipedia? [17:45:01] yes [17:45:10] Okay. [17:45:18] I'll try it too [17:45:29] but other proxies should work too.. can you describe what your problems were? Couldn't you log in? [17:45:30] It's opening..... [17:45:53] IP editing is blocked of course.. but also editing with accounts? hm... well, may be [17:46:01] try opera [17:46:12] When the login page appears [17:46:40] but when i put the user name and password, after that screen is blank [17:46:48] hm.. strange .. okay [17:47:17] Oh, I use Opermini on my cellphone. [17:47:23] you can download here http://www.opera.com/browser/download/ [17:47:28] I'll try it here [17:48:58] I tried Wikipedia on operamini on my cell. The result was the same. [17:49:11] is turbo enabled? [17:49:17] look in the setttings of operamini [17:49:19] But i'll give it a try here [17:49:47] ah, forget it.. mini always has "turbo" enabled [17:52:36] enabled the turbo [17:52:48] The site got stuck on 61% [17:52:56] it is an icon on the left side of the statusbar [17:53:03] hmm [17:53:42] looks like I have to change the ISP after all [17:53:57] that is really strange that opera turbo doesn't work [17:54:28] do a right click -> configure [17:54:40] switch it "on" - not to "automatic" [17:55:21] it says 4/9 elements and then the website is stuck [17:55:33] did you went to the settings? [17:55:35] Yes I turned it to "on" [17:55:38] ok [17:55:57] have no other clue then.. don't know what is wrong then [17:56:19] did you tried the https-version to circumvent proxies on the isp-side? [17:56:55] No [17:56:58] DaBPunkt: the "turbo" is ssl - afaik [17:57:06] ssl to sweden ;) [17:57:28] I really don't know what that means DaBPunkt [17:57:39] Snake_lover: oopen that page: https://en.wikipedia.org/wiki/Main_Page [17:57:45] what do you see? [17:58:33] WOW!!!!!!!!!!! [17:58:49] it's opening up much faster then google [17:59:03] treffer :) [17:59:19] Hey what did you do to the link? [17:59:23] s [17:59:29] https instead of http [17:59:31] https [17:59:40] OOO i see [17:59:50] so this boosts up the speed then? [17:59:54] either your ISP or your country is nasty [17:59:55] :D [17:59:59] HTTPS thing [18:00:11] Country is ;) [18:00:29] What is this thing really? [18:00:30] https usually does slow down - except in interception of http traffic by the ISP [18:00:43] https sets up an encrypted connection to wikipedia [18:00:57] In simple words please :) [18:01:01] in theory no one (also not your ISP or country's agencies) can look into [18:01:30] Okay [18:01:41] well, then - deinstall opera again ;) [18:02:13] Okay [18:02:38] Thanks guys [18:02:48] I really appreciate it [18:02:51] https://en.wikipedia.org/wiki/HTTPS#Difference_from_HTTP here is the wikipedia expanation [18:03:02] I will leave a note in the bugreport [18:03:18] strange that no one tried https before.. :D [18:03:36] hehe [18:03:59] So it's safe using HTTPS? [18:04:55] Snake_lover: yes, really safe [18:05:02] Okay [18:05:05] except your country or ISP forbids it ;) [18:05:15] usually you use it for home banking [18:05:29] or credit card transactions when shopping [18:05:44] maybe you saw it there before [18:06:06] Meaning it's for extra security if money is involved [18:06:13] yes [18:06:43] Okay [18:06:49] Thanks a million Saibo [18:07:13] Snake_lover: thank also DaBPunkt! :D [18:07:38] your welcome [18:07:39] DaBPunkt Thanks !!! [18:07:43] :) [18:07:55] Bye guys [18:08:00] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [18:08:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [18:08:09] Snake_lover: a hint: you can also acccess google via https [18:08:21] and many other sites [18:08:41] https://encrypted.google.com/ [18:09:04] Well all other sites were working fine except for Wikipedia, but now it's also fine [18:09:08] I [18:09:15] may be faster ;) [18:09:25] I'll keep the S in mind ;) [18:09:35] okay, bye [18:09:42] :) [18:10:05] we should default to https [18:10:12] why? [18:10:16] would have prevented that problem :D [18:10:28] that's a massive expenditure of money for basically no security [18:10:36] unless you mean defaulting to HTTPS for logged in users [18:10:45] yes, at least that [18:10:56] that user is from pakistan [18:11:06] and apparently something is fishy at his connection [18:11:18] at least http stucks but https works [18:11:31] I would not wonder if there is some interception going on [18:11:44] New patchset: Mark Bergsma; "Attempt to unbreak LVS in eqiad" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2459 [18:12:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2459 [18:12:12] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2459 [18:12:12] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2459 [18:14:09] while we are at that topic (SSL and man in the middle): interesting http://www.h-online.com/security/news/item/Mozilla-considers-removing-Trustwave-CA-1430998.html and https://bugzilla.mozilla.org/show_bug.cgi?id=724929 [18:15:18] New patchset: Mark Bergsma; "Unbreak LVS by allocating eqiad swift svc ip" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2460 [18:15:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2460 [18:16:36] werdna: a short question: you did mention "that's a massive expenditure of money" - was that a fact? Is it really that more expensive? [18:17:20] Saibo: Yes, HTTPS has substantial overhead [18:17:24] there's no point using it for reading [18:17:43] oh, there is [18:17:43] Thats untrue. [18:17:54] no one need to know what I read [18:17:57] (that there is no point in using it for reading) [18:18:17] I know there is overhead - the question is: how much in numbers? [18:18:28] how much more money does it cost in percent [18:18:30] I don't have numbers of course [18:18:33] gmaxwell: Unless you're in a shady country where they don't like WP [18:18:33] ok [18:18:40] The articles people read on wikipedia are often private and personal. You may not want your family to know that you were reading about genital warts or your state to know you were reading about democracy. [18:18:40] but I don't think it's worth forcing everybody to use HTTPS [18:18:53] gmaxwell: your family is much more likely to know that by reading your browser history [18:18:58] :D [18:19:11] if you're clearing your browser history then you probably know enough to use HTTPS [18:19:27] werdna: They may be— though private mode brosing is a standard feature, and many home routers have URL logging. [18:19:53] mmm, my point is that it's not worth it to do it by *default* [18:19:53] In any case— it's not correct to say it provides no value, it might be correct to say that its not worth mandating it for logged out users. [18:20:15] I think you're getting bogged down in the semantics of my sentence "there's no point" [18:20:38] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2460 [18:20:38] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2460 [18:20:42] Though whenever its not mandated its subject to downgrading attack, e.g. just block SSL. Done. [18:21:12] there may also be negative sides for logged-out users. E.g. if https traffic is blocked in some network... That would also be a downside with a default [18:21:48] werdna: I'm only asking you make yourself clear then. There are many people who actually think that there is no reason to treat reading traffic as private. [18:21:50] RECOVERY - Puppet freshness on lvs1006 is OK: puppet ran at Thu Feb 9 18:21:39 UTC 2012 [18:22:02] fairy nuff [18:22:28] It's also worth keeping in mind that the notion that the user will 'know' to use HTTPS is a bit weak— simply because the reader often can't reason about their exposure, since interception is transparent to them. [18:23:26] E.g. how many of our readers know that ISPs is supposidly free countries have intercepted our traffic in order to insert fake 404 pages to keep people from seeing an article? (and presumably logging all they viewed at the same time) We know that— our readers don't. [18:23:34] s/is/in/ [18:24:20] btw: that channel is publicly logged.. ;) [18:24:20] RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Thu Feb 9 18:23:50 UTC 2012 [18:24:28] *thumbs down* [18:24:30] :D [18:24:56] meh. At least you _know_ it's logged. It's prudent to assume all irc channels are logged. [18:25:15] heh [18:25:34] In any case, I think the benefits of mandating ssl for logged in users are pretty clear. [18:25:55] If they're inadequately clear, I'll sniff some wifi at a meetup and demonstrate on the enwp mainpage if you like. :) [18:26:05] hrhr [18:27:02] we have some dev work we need to do in order to make mandatory https for logged in users work well, but we're tentatively planning on doing that later this year. [18:27:28] there are a few, but they're still not persuasive I don't think. I realise this is an issue of privilege, but there's a reason that 99% of sites on the internet don't mandate HTTPS for casual readers [18:27:52] robla: I wouldn't make it forced - but suggested more than it is now (e.g. by defaulting to that) [18:28:16] I mean.. some may not be able to use https (network policy-based blocks) [18:28:26] werdna: their customers are the product, and their privacy isn't a real concern. Public libraries opposed the patriot act, video rental places did not. :) [18:28:53] Ryan_Lane and RoanKattouw have thought through a lot of this, and I know Ryan has a spec trapped in his head :) [18:29:26] gmaxwell: so, do public libraries force their users to use HTTPS? [18:29:31] werdna: NDAs limit my ability to discuss specifics, but we can expect a significant and growing fraction of cleartext http web traffic in the US to be actively intercepted, modified, and monitored now and in the coming years. [18:29:54] werdna: they don't offer books over the internet for the most part. I suspect many would if they did and they considered this. [18:30:31] * robla disappears into next meeting [18:30:33] gmaxwell: urgs.. we live in dark times... [18:30:58] btw: does wikimedia has servers serving in freenet? ;) [18:31:19] In any case, I'm OKAY with the idea that we're not past the balance for supporting it for casual readers. Things have to move one step at a time. [18:31:21] Saibo: that is not dead? [18:31:29] DaBPunkt: hm.. didn't hear than [18:31:31] that [18:31:59] DaBPunkt: 17 August 2011 it was alive - according to enwp [18:32:22] hah. Freenet isn't dead.. But it doesn't work especially well— I don't think its really interesting. [18:32:30] Saibo: "dead" in the meaing: "nearly nobody use it", not in a technical one [18:32:48] one → sense [18:32:50] * Saibo was just over-over-estimating the facts that you mentioned, gmaxwell [18:32:52] I think it's pretty much just child porn these days, right? [18:33:28] werdna: better do not make claims if do not know ;) [18:33:37] like the https cost thing [18:34:05] That's not really a claim, so much as a quip [18:34:18] s/claim/somethingelse [18:34:20] :) [18:34:22] Saibo: https is fairly costly, objectively. Sadly— if the design were a bit different it could be a lot cheaper. But .. so? It's also quite valuable. [18:35:12] As the technologically adept we need to consider ways of making systems which are protective of the public and which achieve a good balance. Asking the public to defend themselves, e.g. by reading via freenet, is a tall order, because this stuff can be hard to understand and reason about. [18:35:57] (And besides, anonymity services have sucky performance— they are orders of magnitude more 'costly' than SSL— someone who is only mildly concerned will skip them) [18:36:21] I once tried freenet - some years ago - I know it is very very sucky [18:36:36] nod [18:36:38] but.. it works.. nothing can stop it.. in theory [18:36:42] I fiddled for a while. [18:36:53] and that is quite a really good feeling [18:37:23] I don't know, as a programmer I hate things you can't get rid of. You end up paying technical child support. [18:37:26] Like on LiquidThreads [18:37:29] Publishing wikipedia into freenet would be good for freenet (give it some more ethical uses)— but I think not very relevant for wikipedia or its users. [18:38:36] It wasn't a real suggestion or wish - just a idea related to our discussion [18:38:44] it would be very costly I guess [18:38:54] .. the programming [18:39:51] Nah. I think it would be fairly easy— at least using the HTML dumps, which I think haven't been updated in a long time— and its something anyone could do. [18:41:54] hmm.. ok [18:46:13] http://planet.wmflabs.org/ [18:52:59] www.wmfbbq.com [18:53:23] d'aw [19:00:00] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.380 seconds [19:00:17] who is ekrem? [19:00:47] ToAruShiroiNeko: The server that runs the WAP gateway and irc.wm.o [19:01:30] why call it ekrem? [19:01:56] http://en.wikipedia.org/wiki/Ekrem [19:04:37] I have no idea [19:18:43] AaronSchulz: looking at http://www.mediawiki.org/wiki/SwiftMedia/Issues [19:18:51] I pulled up the image the guy posted about [19:19:06] the files exist in swift, but an ?action=purge doesn't clear them out. [19:19:38] hm. I'm probably just getting teh URL wrong. [19:19:48] no, I got the 'are you sure click ok' button. [19:19:49] hm. [19:19:55] any ideas? [19:20:12] (it looks like it's not a swift-specific thing since they're not getting purged from ms5 either) [19:23:43] * AaronSchulz doesn't know what file is being described [19:23:51] * AaronSchulz works on an thumb.php exploit [19:26:37] nighty =_= [19:30:03] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:23] AaronSchulz: what's the current right way to maintenance scripts from a checkout in my fenari home dir? [19:32:26] !log reedy synchronized wmf-config/CommonSettings.php 'Increase wgCodeReviewMaxDiffPaths' [19:32:28] Logged the message, Master [19:33:11] binasher: http://wikitech.wikimedia.org/view/Het_deploy#Run_a_maintenance_script_on_a_wiki [19:33:47] thanks [19:33:53] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:57] does it matter where the script i'm running is located though? [19:34:12] sort of, where is it? [19:34:32] php-1.18/maintenance or such? [19:34:46] the 1.19 migration is currently in the REL1_19 branch since 1.19wmf hasn't been cut [19:35:33] should i just copy what i need into /home/w/common/php-1.18? [19:35:41] you should port it into 1.18wmf [19:35:44] yeah [19:35:46] or actually merge into 1.18wmf? [19:36:00] I guess merge, since robla dislike uncommitted stuff there [19:36:04] ok [19:46:43] maplebed: you were looking at http://de.wikipedia.org/wiki/Datei:Wasserhose_1884.png ? [19:47:19] yeah. [19:47:48] now I think there's a problem stuck in the squid cache [19:47:58] I manually cleared out both ms5 and swift [19:48:52] purging for that image still isn't working though. [19:48:56] no idea why. [19:48:58] http://upload.wikimedia.org/wikipedia/de/thumb/e/ee/Wasserhose_1884.png/120px-Wasserhose_1884.png [19:49:05] Unexpected error HTTP Error 500: Internal server error [19:50:13] I don't get it, but I don't think it's related to swift. [19:50:14] :( [19:50:39] did it work a few days ago? [19:51:26] no idea. [19:51:26] AaronSchulz: i can't say, just saw it on dewiki village pump [19:51:57] http://de.wikipedia.org/wiki/Datei:Wasserhose_1884.png implies that it was uploaded last June [20:01:38] maplebed or uploaded in 1884 if the filename is any indication [20:01:50] lol [20:04:36] !log started 1.19 schema migrations on seconadry db's (mwscript upgrade-1.19wmf1-1.php --secondary) [20:04:38] Logged the message, Master [20:05:23] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.531 seconds [20:14:40] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 1253 seconds [20:16:20] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 1353 seconds [20:17:55] AaronSchulz, maplebed: Another image: http://de.wikipedia.org/wiki/Datei:UboottypXXI-FullRes.png [20:17:59] also starts with ee [20:20:40] PROBLEM - MySQL Slave Delay on db1017 is CRITICAL: CRIT replication delay 1613 seconds [20:21:14] New patchset: Jgreen; "added community-analytics vhost for aluminium/grosley" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2464 [20:21:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2464 [20:22:00] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 1693 seconds [20:22:10] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 1703 seconds [20:22:11] New review: Jgreen; "add, but don't yet enable apache virtual server definition" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2464 [20:22:11] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2464 [20:30:00] PROBLEM - MySQL Replication Heartbeat on db1017 is CRITICAL: CRIT replication delay 2173 seconds [20:31:17] New patchset: Jgreen; "removing references to puppet:///files/nagios/nrpe_local.fundraising.cfg which was removed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2465 [20:31:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2465 [20:31:48] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2465 [20:31:48] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2465 [20:35:00] New patchset: Demon; "Add option for skipping non-matches since writing extra rules is annoying" [operations/software] (master) - https://gerrit.wikimedia.org/r/2466 [20:35:01] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2466 [20:44:30] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:52] New review: Catrope; "(no comment)" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2466 [20:44:59] New review: Diederik; "(no comment)" [analytics/udp-filters] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2233 [20:45:21] New review: Diederik; "Ok." [analytics/udp-filters] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2182 [20:45:21] New review: Demon; "(no comment)" [operations/software] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2331 [20:45:21] Change merged: Diederik; [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2233 [20:45:21] Change merged: Diederik; [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2182 [20:45:22] Change merged: Demon; [operations/software] (master) - https://gerrit.wikimedia.org/r/2331 [20:45:41] New review: Diederik; "Ok." [analytics/udp-filters] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2232 [20:45:41] Change merged: Diederik; [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2232 [20:46:08] New review: Diederik; "Ok." [analytics/udp-filters] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2234 [20:46:08] Change merged: Diederik; [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2234 [20:46:25] New review: Diederik; "Ok." [analytics/udp-filters] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2235 [20:46:25] Change merged: Diederik; [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2235 [20:49:29] New review: Demon; "(no comment)" [operations/software] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2332 [20:49:29] Change merged: Catrope; [operations/software] (master) - https://gerrit.wikimedia.org/r/2466 [20:49:29] Change merged: Demon; [operations/software] (master) - https://gerrit.wikimedia.org/r/2332 [20:56:00] RECOVERY - MySQL Slave Delay on db1017 is OK: OK replication delay 0 seconds [20:59:30] PROBLEM - MySQL Slave Delay on db1033 is CRITICAL: CRIT replication delay 701 seconds [21:01:20] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 1790 seconds [21:02:30] PROBLEM - MySQL Slave Delay on db1043 is CRITICAL: CRIT replication delay 880 seconds [21:04:40] RECOVERY - MySQL Replication Heartbeat on db1017 is OK: OK replication delay 0 seconds [21:04:50] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 2000 seconds [21:18:30] PROBLEM - MySQL Replication Heartbeat on db1018 is CRITICAL: CRIT replication delay 691 seconds [21:22:40] PROBLEM - MySQL Replication Heartbeat on db1034 is CRITICAL: CRIT replication delay 941 seconds [21:24:00] PROBLEM - MySQL Slave Delay on db1034 is CRITICAL: CRIT replication delay 1021 seconds [21:24:10] PROBLEM - MySQL Replication Heartbeat on db1002 is CRITICAL: CRIT replication delay 1031 seconds [21:24:30] RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay 0 seconds [21:29:05] RECOVERY - MySQL Slave Delay on db1043 is OK: OK replication delay 0 seconds [21:36:42] !log reedy synchronized wmf-config/CommonSettings.php 'Increase wgCodeReviewMaxDiffPaths' [21:36:44] Logged the message, Master [21:37:15] RECOVERY - MySQL Slave Delay on db1033 is OK: OK replication delay 0 seconds [21:39:23] AaronSchulz: Seems like the pics worked before the change [21:40:25] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [21:40:35] PROBLEM - MySQL Slave Delay on db1002 is CRITICAL: CRIT replication delay 929 seconds [21:42:03] New patchset: Ottomata; "Renaming the concept of variables to 'traits'. Allowing trait_sets to be specified so that we don't record HUGE amounts of data." [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2467 [21:42:04] New patchset: Ottomata; "Adding loader.py - first hacky loader, just so we can get some data into mysql to work with." [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2468 [21:45:25] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [21:46:35] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay 0 seconds [21:47:15] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [21:51:17] New patchset: Ottomata; "device_pipeline.py - comments about hackyness" [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2469 [21:55:55] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: CRIT replication delay 774 seconds [22:02:52] New patchset: Lcarr; "Upping concurrent nagios service checks This should avoid false alarms Also removing bonding.conf which is irrelevant" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2470 [22:03:55] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 0 seconds [22:05:55] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds [22:15:15] RECOVERY - MySQL Slave Delay on db1002 is OK: OK replication delay 0 seconds [22:18:25] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.907 seconds [22:19:15] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [22:30:39] maplebed: var_dump( $file->getWidth() * $file->getHeight() > $wgMaxImageArea ); [22:30:41] bool(true) [22:30:59] huh? [22:31:00] only image/jpeg can get around that by using imagemagick, not pngs [22:31:27] maplebed: I messed around in eval.php with that file [22:31:48] PngHandler::normaliseParams() fails due to that [22:31:56] did this file really work before? [22:32:34] Steef389 said it did... [22:32:48] but isn't what you are describing something that would affect all pngs? [22:32:51] why just this one? [22:32:58] only really large ones [22:33:22] i posted another one above: http://de.wikipedia.org/wiki/Datei:UboottypXXI-FullRes.png [22:33:28] thumb path has also ee [22:33:37] also a really large file [22:34:00] (which is more in line with what AaronSchulz is talking about) [22:35:10] $wgMaxImageArea is 12500000 fyi [22:35:41] TWELVE MILLION FIVE HUNDRED THOUSAND DOLLARS [22:35:42] 12.5 megapixel [22:35:50] not so large these days. [22:36:10] http://www.nikonusa.com/Nikon-Products/Product/Digital-SLR-Cameras/25480/D800.html [22:36:17] New patchset: Jgreen; "enabled community-analytics virtual server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2471 [22:36:38] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2471 [22:36:38] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2471 [22:38:12] yeah, so that certainly could use a more descriptive error message [22:38:58] I thought error messages supposed to be 256bit encrypted [22:39:01] maplebed: VipScaler was supposed to have handled such pngs by now, but the vips code had a performance bug where it loaded the whole into RAM anyway [22:39:04] you know the VIPS author has just made a patch allowing memory-efficient scaling of PNGs [22:39:27] so maybe we will have that soon [22:40:29] maplebed: do you have that camera? :) [22:40:29] Does nad hang out here? [22:40:44] AaronSchulz: only in my dreams. [22:41:12] (though I'm actually very happy with my existing camera, my inherent geek/gadget desire hounds me) [22:43:49] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2470 [22:43:50] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2470 [22:45:17] AaronSchulz: files with d are served from swift too? [22:45:28] all thumbs are [22:45:52] this png is bigger, but thumb works: http://de.wikipedia.org/wiki/Datei:Logo_von_Anno_1404_Venedig.png [22:46:35] it's area is not bigger [22:46:43] that's where the limit is enforced [22:46:43] just saw [22:47:15] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 0.130 seconds [22:56:57] New patchset: Ottomata; "Movied wurfl.py file" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2472 [22:56:58] New patchset: Ottomata; "More documenation, added tests for AccessLogPipeline methods." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2473 [22:56:59] New patchset: Ottomata; "device_pipeline.py - getting rid of debug() function, using main()." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2474 [22:57:00] New patchset: Ottomata; "Hacky first work on loader classes." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2475 [22:57:02] New patchset: Ottomata; "base.py - adding schema in comments. Got lots of work to do to make this prettier" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2476 [22:57:03] New patchset: Ottomata; "Renaming the concept of variables to 'traits'. Allowing trait_sets to be specified so that we don't record HUGE amounts of data." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2477 [22:57:04] New patchset: Ottomata; "Adding loader.py - first hacky loader, just so we can get some data into mysql to work with." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2478 [22:57:05] New patchset: Ottomata; "device_pipeline.py - comments about hackyness" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2479 [22:59:33] AaronSchulz: so is it settled? it's just the risizing bug and nothing to do with swift or the migration? [23:00:19] yeah, well more of "specific limitation" that bug [23:00:20] New patchset: Ottomata; "pipeline/user_agent.py - adding comment that this file should not be used" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2480 [23:00:53] ok. thanks. [23:01:11] so AaronSchulz my response here is correct? https://www.mediawiki.org/wiki/SwiftMedia/Issues [23:01:30] looks sane [23:01:44] tyvm! [23:02:48] so the limit is 12.5 mega-px? [23:03:10] looks that way. [23:03:14] k thy [23:03:17] (and only applies to .png) [23:03:24] jpgs will be fine. [23:12:43] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [23:12:53] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: CRIT replication delay 1661 seconds [23:13:26] New patchset: Diederik; "Adding dependency list for virtualenv" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2481 [23:14:13] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [23:14:28] New review: Ottomata; "(no comment)" [analytics/reportcard] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2481 [23:16:23] PROBLEM - MySQL Slave Delay on db1002 is CRITICAL: CRIT replication delay 1871 seconds [23:17:34] New patchset: Lcarr; "Increasing the max concurrent checks on nagios to 128" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2482 [23:19:03] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:34] Deploying Swift for thumbnails this week (Feb6-9) ...and ping maplebed [23:19:36] New review: Lcarr; "Since 96 has no real impact on the box memory and cpu-wise, trying 128" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2482 [23:19:37] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2482 [23:20:11] maplebed: http://www.google.de/search?q=%22%26%22+site:wikimedia.org+inurl%3Afile [23:20:38] hi [23:20:50] what's that search showing? [23:20:54] maplebed: Hi, When file-name contains "&", thumb fails. [23:21:05] New review: Diederik; "Ok" [analytics/reportcard] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/2481 [23:21:05] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2481 [23:21:10] oh, interesting. [23:21:13] * maplebed looks [23:21:18] Some are still chached but when you purge you see it. [23:21:38] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2478 [23:21:42] AaronSchulz: ^^^^ [23:22:53] no we are talking encoding :) [23:23:00] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2479 [23:23:12] that'd be my guess too. [23:23:25] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2477 [23:23:30] *now [23:23:35] man, why can't any of these names with an & be less than a million characters long?! [23:23:37] ::sigh:: [23:23:47] ah. [23:23:48] http://commons.wikimedia.org/wiki/File:F1_2010_%26_2011_rear.jpg [23:23:50] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2480 [23:24:06] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2476 [23:24:18] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2475 [23:24:31] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2474 [23:24:33] * maplebed mutes gerrit. [23:24:59] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2473 [23:25:23] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2472 [23:25:23] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2474 [23:25:23] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2473 [23:25:24] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2472 [23:25:39] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2469 [23:25:52] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2468 [23:26:28] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2467 [23:26:29] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2469 [23:26:29] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2468 [23:26:29] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2467 [23:26:52] AaronSchulz: the & is showing up in the swift logs as %2526 [23:28:13] PROBLEM - Disk space on ms1002 is CRITICAL: DISK CRITICAL - free space: /export/upload 762855 MB (3% inode=98%): [23:30:28] probably something in rewrite.py [23:30:33] fastinating. [23:30:41] running tcpdump on the proxy [23:30:48] +.u....HGET /wikipedia/commons/thumb/5/5d/F1_2010_%26_2011_rear.jpg/169px-F1_2010_%26_2011_rear.jpg HTTP/1.0 [23:30:48] ...J....GET /wikipedia/commons/thumb/5/5d/F1_2010_&_2011_rear.jpg/169px-F1_2010_&_2011_rear.jpg HTTP/1.1 [23:30:58] I think the first one is coming in and the second is going out, but I'm not positive. [23:31:19] means it's coming in as a %26, getting logged as %2526, and getting handed off to ms5 as & [23:31:30] how is swift at 100% doing? [23:31:50] apergos: alright. it feels slow when you load commons' Special: New Files [23:32:05] hmm [23:32:07] * apergos tries it [23:32:16] hrm... I don't think I heard my -commons people complain [23:32:23] so that's good, right? [23:32:24] but it's also getting about an 80% miss rate, so with the new hardware coming in next Tuesday and some time to fill its disks, it might pick up. [23:32:39] but at the moment, it looks like rillke may have found us a bug. [23:32:41] :) [23:32:45] maplebed: 80% miss? [23:33:05] hexmode: i.e. 80% of the thumbnails being requested don't already exist in swift. [23:33:06] maplebed: really? did he file something in bz? [23:33:10] one blank thumb [23:33:15] the resst came in on time though [23:33:16] that number comes from here: http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&m=&tab=v&vn=swift [23:33:22] hexmode: no, pinged me here. [23:33:37] what was the bug? [23:33:45] (as I suggested in the various pumps and emails and blog post. :) ) [23:33:53] files with an & in the name aren't getting thumbnailed. [23:34:03] EG http://upload.wikimedia.org/wikipedia/commons/thumb/5/5d/F1_2010_%26_2011_rear.jpg/169px-F1_2010_%26_2011_rear.jpg [23:34:06] sure, if you said do it here, that's good [23:34:34] but I was also thinking I would see them in bz [23:34:53] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.363 seconds [23:35:02] but if rillke only gave you one bug, then that is practically perfectt [23:35:20] there was one other reported but it turned out to be unrelated to swift. [23:35:29] * hexmode is struggling in console-only mode [23:35:42] awesome :) [23:36:39] AaronSchulz: are you looking at rewrite? [23:37:01] * AaronSchulz is a bit distracted [23:37:34] I would appreciate any help you can give, considering it's a live bug. [23:41:46] * AaronSchulz stares at req.path_info = "/v1/%s/%s/%s" % (self.account, container, urllib2.unquote(obj)) [23:43:28] this is the line I was looking at: upcopy = opener.open(reqorig.url) [23:43:39] trying to see if there was a .encode() method to webob objects or something like that [23:43:50] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:16] huh. I'm confused by something. 5/5d/Wilts_&_Dorset_138_HF05_HXH_rear.JPG/120px-Wilts_&_Dorset_138_HF05_HXH_rear.JPG currently exists in swift and loads just fine, but you can't create new ones. [23:48:26] How'd it get in there if not by the same creation path? [23:48:54] (there are, in fact, tons of images with & in the name.) [23:57:37] New patchset: Lcarr; "More nagios tweaking upped parallel service checks to 192 and now will process them every 9 seconds instead of every 10" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2483 [23:59:40] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2483 [23:59:40] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2483