[01:54:06] <Elsie>	 ori-l: Any chance I could get you to pastebin "show tables;" on enwiki?
[01:56:14] <Krenair>	 Elsie, I'm guessing you want it from production not labs' enwiki_p?
[01:56:23] <Elsie>	 Yeah. :-)
[01:59:29] <ori-l>	 will it get me into trouble?
[01:59:34] <ori-l>	 why do you need it?
[01:59:50] <Elsie>	 I did user prefs clean out bugs today.
[01:59:57] <Elsie>	 I was going to do table clean out bugs tomorrow.
[02:00:07] <Elsie>	 I doubt it'll get you into trouble.
[02:00:11] <ori-l>	 alright, give me a moment
[02:02:00] <Krenair>	 lol ori-l, I doubt any table names are sensitive
[02:02:29] <Elsie>	 There's apparently a hashs table.
[02:02:36] <Elsie>	 No idea why or how.
[02:02:46] <ori-l>	 Krenair: they're case-sensitive!
[02:03:00] <Elsie>	 sbm
[02:06:07] <ori-l>	 Elsie: https://dpaste.de/vIaGX/raw/
[02:07:02] <Elsie>	 ori-l: Thanks!
[02:09:44] <Elsie>	 I think some of these tables should be dropped.
[02:09:51] <Elsie>	 But I'm not sure it's my place to say so.
[02:13:41] <ori-l>	 the cost of keeping them around is often close to nil if they are small, and in general unnecessary maintenance on production databases is to be avoid, but it's probably good to have a devil's advocate agitate for periodic housecleaning
[02:14:12] <ori-l>	 i don't know that that's a winning argument, but you could ask
[02:14:15] <ori-l>	 * avoided
[02:14:29] <Elsie>	 Well, some of them are also clearly old and broken.
[02:14:35] <Elsie>	 Because they're named _old or broken_.
[02:14:43] <Elsie>	 I'm not sure what the point of keeping them around forever is.
[02:14:57] <Elsie>	 Like, user_old is from around 2005, I think.
[02:15:02] <Elsie>	 Based on the number of rows.
[02:15:24] * ori-l  shrugs

[02:16:23] <Elsie>	 I'm not sure I'd call the cost negligible.
[02:16:32] <Elsie>	 Given that we have to replicate these tables.
[02:16:41] <Elsie>	 And probably back them up.
[02:16:54] <Elsie>	 And store them, though storage _is_ cheap.
[02:17:00] <Elsie>	 I dunno.
[02:17:04] <Elsie>	 I'll file the bugs.
[02:17:10] <p858snake|l>	 file all the bugs
[02:17:11] <Elsie>	 They're valid bugs, I just don't expect anyone to act on them.
[02:17:15] <p858snake|l>	 closing bugs is cheap
[02:17:28] <Elsie>	 Rarely.
[02:17:42] <p858snake|l>	 only when your involved they aren't
[02:17:58] <Elsie>	 There are also core tables that are like...
[02:18:07] <Elsie>	 For example, hitcounter.
[02:18:28] <p858snake|l>	 no one talks about that stepchild of a table >.>
[03:01:37] <Elsie>	 https://meta.wikimedia.org/wiki/Requests_for_comment/Petition_of_HTTPS_default
[08:46:30] <cff>	 Is there an API endpoint for Wikipedia that can give back a generator for all articles that have a main coordinate?
[14:37:05] <Romaine>	 is anyone aware of cache problems (of something what looks ike that)?
[14:38:52] <Reedy>	 Can you be any more vague?
[14:40:17] <Romaine>	 in the past days we experience somethig what seems to be caching problems, on particular one page on nl-wiki were many times an earlier version of that page is shown instead of the current version
[14:40:56] <Romaine>	 this page is in use to request admin tools
[14:41:09] <Romaine>	 https://nl.wikipedia.org/wiki/Wikipedia:Aanmelding_moderatoren
[14:41:27] <Romaine>	 I just opened that page again and again an earlier evrsion is shown
[14:41:56] <Romaine>	 even while I have seen a more recent version yesterday
[14:42:02] <Romaine>	 a lot of users complain about this
[14:42:12] <Romaine>	 Reedy: is this concrete enough?
[14:42:56] <valhallasw>	 Romaine: I get the most recent revision. I assume you are logged in?
[14:43:02] <Romaine>	 yes
[14:43:20] <Romaine>	 what is the top section of that page?
[14:43:27] <Romaine>	 top bar
[14:43:29] <valhallasw>	  Actueel:
[14:43:30] <valhallasw>	  0 peilingen  |  0 stemmingen  |  1 moderatoraanmelding  |  4 etalageaanmeldingen
[14:43:48] <Romaine>	 yes, that is the right version
[14:43:54] <valhallasw>	 and «zaheer·12a»'s vote is shown
[14:44:06] <valhallasw>	 (#85 voor dqfn13)
[14:44:16] <Romaine>	 but the strange thing is that on verious moments I do not get the most recent version
[14:44:40] <valhallasw>	 of the page or of the template?
[14:44:51] <Romaine>	 page
[14:45:00] <valhallasw>	 so also newer votes do not show up?
[14:45:04] <Romaine>	 see the history
[14:45:24] <Romaine>	 yes, some votes are missing when reading a page
[14:45:47] <Romaine>	 but when editing those are back or aren't back
[14:46:09] <valhallasw>	 so you even sometimes get an old version in the editor? That's very strange.
[14:46:15] <Romaine>	 twice already someone deleted a lot of votes as he got an older version on his screen and edited that one
[14:46:30] <Romaine>	 it is not me only, several users have
[14:46:36] <Romaine>	 see also the history of the page
[14:47:08] <Romaine>	 * there are moments people get the most recent page
[14:47:33] <Romaine>	 * there are moments people get an older version, but with clicking edit they edit the current page
[14:48:01] <Romaine>	 * there are moments people get an older version, and click edit and also edit in an older version (with loss of votes)
[15:09:41] <Romaine>	 Reedy / valhallasw: any idea what this causes and/or how this can be solved?
[15:14:27] <Romaine>	 Reedy / valhallasw: Help! I just reopened that page again and again I got the older version of it, instead of the current version
[15:15:40] <Romaine>	 I also noticed that templates that are inserted in the time between the older version shown and the current version aren't always listed in https://nl.wikipedia.org/wiki/Speciaal:VerwijzingenNaarHier/Wikipedia:Mededelingen/Actueel
[15:25:12] <Romaine>	 Reedy / valhallasw:I created a bug for it: https://bugzilla.wikimedia.org/show_bug.cgi?id=52853
[15:34:56] <valhallasw>	 Romaine: kan je daar nog bij posten welke server je pagina heeft gerenderd? right-click, view source, dan onderaan iets als '<!-- Served by mw1213 in 2.008 secs. -->'
[15:35:11] <Romaine>	 zal ik doen
[15:35:46] <Romaine>	 <!-- Served by mw1058 in 1.962 secs. -->
[15:36:05] <Romaine>	 ^^ with showing date on the page itself: 10 aug 2013 om 19:30
[15:36:11] <Romaine>	 (as last version)
[15:38:06] <valhallasw>	 ah!
[15:38:12] <valhallasw>	 it seems to be http/https related
[15:38:29] <valhallasw>	 or at least: I get a new version over https, an old one over http
[15:38:47] <Romaine>	 I get also old versions with https
[15:39:15] <valhallasw>	 oh, you're right. It changes on a refresh
[15:40:30] <Romaine>	 it keeps showing or the current version or one particular older version: 10 aug 2013 om 19:30
[15:41:17] <valhallasw>	 hm.
[15:42:38] <Romaine>	 I refreshed the page 20 times, 8 out of 20 I got the version of 10 Aug
[15:42:49] <Romaine>	 12 out of 20 the most recent revision
[15:42:54] <Romaine>	 randomly
[15:48:21] <Romaine>	 valhallasw: I added some <!-- Served by
[15:48:21] <Romaine>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=52853
[15:59:56] <valhallasw>	 andre__: could you also grep the files for 'laatst bewerkt op'?
[16:00:19] <valhallasw>	 I actually get a Last-Modified: Wed, 14 Aug 2013 15:37:57 GMT  for a page that in HTML states 'laatst bewerkt op 10 aug 2013 ....'
[16:01:37] <andre__>	 valhallasw, same result, all Aug14
[16:01:45] <andre__>	 you might want to slack in #wikimedia-operations
[16:01:55] <andre__>	 I've brought up the issue there
[16:02:01] <andre__>	 because this sounds like ops territory
[16:02:06] <valhallasw>	 right
[16:02:08] <valhallasw>	 Romaine: ^
[16:02:11] <andre__>	 but no answers yet :-/
[16:02:17] <andre__>	 and I got to leave in ~30min
[16:16:09] <svick>	 apergos, parent5446: hello
[16:16:24] <apergos>	 hello
[16:19:30] <parent5446>	 Hey
[16:19:37] <svick>	 so, today i'm continuing working on saving diff dumps
[16:19:54] <svick>	 then it's reading and applying them
[16:21:03] <apergos>	 sweet
[16:21:19] <apergos>	 you're puttig randall out of business you know (you saw the email thread?)
[16:22:13] <svick>	 yeah, i read it
[16:23:27] <apergos>	 so I have a bug to report, or a pebcak problem, we'll see
[16:23:27] <svick>	 and you know i planned this long before i knew about his work :-)
[16:23:40] <svick>	 ok, what is it?
[16:24:16] <apergos>	 no I think it's a good thing he's going to be put out of business, the program he has is a rought stopgap (which is cool) but it is just that, a stopgap
[16:24:17] <apergos>	 so
[16:24:26] <apergos>	 ./idumps c tenwiki-20130503-pages-meta-history.xml sh stubs-history.out
[16:24:40] <apergos>	 oh nm
[16:24:41] <apergos>	 grr
[16:24:46] <apergos>	 brain dead again.
[16:25:06] <apergos>	 oh no
[16:25:14] <apergos>	 not brain dead, unless I have the order of the args wrong
[16:25:17] <apergos>	 so the creation runs
[16:25:18] <apergos>	 then
[16:25:19] <apergos>	 ./idumps r stubs-history.out  blot.xml
[16:26:03] <apergos>	 and this is after removing the stubs-history.out first, just to make sure
[16:26:11] <apergos>	 so the read core dumps
[16:26:16] <apergos>	 terminate called after throwing an instance of 'std::out_of_range'
[16:26:17] <apergos>	   what():  basic_string::at
[16:26:17] <apergos>	 Ακυρώθηκε (core dumped)
[16:26:40] <svick>	 let me try it out
[16:26:58] <apergos>	 sure
[16:27:09] <apergos>	 I had rebuilt idump just before all that
[16:27:28] <apergos>	 you don't have to trry it this second, just stick it in your queue someplace
[16:28:52] <svick>	 it's possible that this is a bug that was there before, but it just didn't show until i switched to at(); i'll know more when i look at it
[16:28:57] <apergos>	 sure
[16:29:41] <apergos>	 well there's a good reason for the switch :-D
[16:30:13] <svick>	 one thing i was thinking about is identifying dumps; with diff dumps, i want to make sure a user can't apply diff to the wrong dump
[16:30:59] <svick>	 i guess basically the same system that's in place now will work: wiki name (e.g. enwiki) + date + dump kind (e.g. stub-history)
[16:31:40] <svick>	 where date will be treated as a string that has to be lexicographically bigger for each new version
[16:32:01] <apergos>	 so thinking about applying these
[16:32:13] <apergos>	 in a dream future where we produce diff dumps every day
[16:32:20] <apergos>	 (so not going to happen but suppose we could)
[16:32:36] <apergos>	 someone might download once a week but not want to have to run the script to apply, 7 times in a row
[16:32:45] <apergos>	 or 6 I guess
[16:33:03] <apergos>	 I expect they would rather want to run the script once and give it 6 files plus the full
[16:33:52] <apergos>	 his might be something to keep in mind
[16:33:55] <apergos>	 *this
[16:34:02] <svick>	 i don't see a problem with that, it would just apply the files one after another
[16:34:44] <svick>	 and Randall's suggestion might be useful too: produce daily *and* weekly (and monthly) diffs
[16:34:50] <svick>	 for current dumps
[16:35:31] <apergos>	 well I wonder if one could be cleverer about applying them if one had several of them
[16:35:54] <apergos>	 and yes we need to think about whether we could do weekly or monthly diffs etc
[16:36:15] <apergos>	 or even dailies... that would be a whole new world there
[16:38:08] <apergos>	 I guess it's going to depend on how they get applied... can't wait to see!
[16:38:33] <svick>	 you're right, if there were multiple changes, it could apply just the most recent one; that would probably speed things up
[16:40:28] <svick>	 i'll try not to make you wait too long :-)
[16:41:09] <apergos>	 :-)
[16:41:41] <apergos>	 so another option is 
[16:41:47] <apergos>	 instead of producing a weekly
[16:42:01] <apergos>	 being able to combine the 6 dailies into a weekly, if it's cheaper
[16:42:08] <apergos>	 (which I can always hope)
[16:42:32] <apergos>	 because in theory we have all the info right there, and that saves wear and tear on the dbs to
[16:42:32] <apergos>	 o
[16:43:55] <svick>	 hmm, that might be faster, but it's alos more complicated
[16:44:09] <apergos>	 yep
[16:44:15] * apergos  lookks hopeful

[16:44:20] <apergos>	 *looks
[16:44:41] <svick>	 and you can create daily and weekly at the same time, which means it will read the db just once
[16:45:09] <apergos>	 I'm not quite following that
[16:45:20] <apergos>	 you create a new daily every day
[16:45:34] <Clinteger>	 hmm
[16:45:35] <apergos>	 on one of thoe days you create a daily and a weekly (maybe at the same time)
[16:45:49] <apergos>	 that weekly means you have to go request all those texts you already got on days 1-5
[16:45:51] <Clinteger>	 does it make sense that i am having routing problems to bits.wikimedia.org but not any other wikimedia sites? :|
[16:45:58] <svick>	 well, you can do something like ./idumps u ... sc oldDaily.id sc oldWeekly.id
[16:46:16] <svick>	 which will update both dumps at the same time, reading from DB only once
[16:46:44] <svick>	 hmm, you're right
[16:47:02] <apergos>	 keep it in the back of your mind as you're working out how the apply will work
[16:47:03] <svick>	 though i could get the texts from the old daily
[16:47:30] <apergos>	 it doesn't have to be there in the first production-viable code
[16:47:35] <svick>	 right
[16:47:56] <apergos>	 but if you plan now to allow for it later you won't get backed into a corner by some design choice now
[16:48:20] <apergos>	 and yes, prefetch is always an option
[16:48:28] <svick>	 yeah, i'll keep that in mind
[16:50:02] <apergos>	 anything else on your mind? 
[16:51:07] <svick>	 i think that's it
[16:51:20] <parent5446>	 I don't have anything either.
[16:52:03] <apergos>	 ok, in that case see everyone tomorrow
[16:52:03] <svick>	 see you tomorrow then, and have a nice day
[16:52:08] <apergos>	 yep you too
[17:13:33] <chrismcmahon>	 manybubbles: check your pm? 
[17:13:50] <manybubbles>	 reading!
[19:42:45] <facundobatista>	 jeremyb, ping! I finally have data for the strange "403 Forbidden" error
[19:43:03] <jeremyb>	 facundobatista: hola
[19:43:10] <jeremyb>	 pastebin or?
[19:43:10] <facundobatista>	 jeremyb, hola :)
[19:43:23] <facundobatista>	 jeremyb, the response body is http://linkode.org/l/Y1EqnvtD4T2eJuj745ium3 , the headers are http://linkode.org/l/1osjlOGy3twJDfwYrwO4Y6
[19:43:49] <facundobatista>	 don't see nothing too useful :/
[19:44:05] <jeremyb>	 hah, mailto:webmaster?...
[19:44:07] <jeremyb>	 useful
[19:44:09] <facundobatista>	 s/nothing/anything/
[19:45:19] <jeremyb>	 facundobatista: so, still inconsistent and no patterns appeared?
[19:45:30] <jeremyb>	 i.e. non-deterministic
[19:45:52] <jeremyb>	 the localhosts in the headers look wrong to me
[19:46:18] <facundobatista>	 jeremyb, this is the first one, I rollouted code to production 1h ago
[19:46:47] <jeremyb>	 facundobatista: do you know if you're hitting amsterdam or virginia or where?
[19:46:53] <jeremyb>	 eqiad or pmtpa or esams
[19:47:05] <jeremyb>	 should be in reverse DNS of the IP you connect to
[19:47:16] <jeremyb>	 also, where are your params? these are POSTs right?
[19:47:36] <facundobatista>	 jeremyb, this is the response
[19:47:48] <jeremyb>	 right
[19:48:23] <jeremyb>	 you're certain there's no proxy on your end? i.e. the squid injecting those headers isn't your squid
[19:48:49] <jeremyb>	 facundobatista: anyway, POST params pls
[19:48:59] <facundobatista>	 jeremyb, a proxy between my code and wikipedia servers, you mean? probably yes
[19:49:21] <jeremyb>	 facundobatista: parse error
[19:49:36] <jeremyb>	 facundobatista: probably yes what? :)
[19:49:44] <facundobatista>	 jeremyb> you're certain there's no proxy on your end? <-- what do you mean "my end"?
[19:50:05] <jeremyb>	 facundobatista: a proxy between your client and the public internet
[19:52:17] <facundobatista>	 jeremyb, probably there is one
[19:52:45] <facundobatista>	 jeremyb, the query that got that result was a GET: http://nb.wikipedia.org/w/api.php?action=opensearch&limit=10&format=xml&search=terminal
[19:53:28] <jeremyb>	 grrrrr, i think my local logging was more broke than i realized (ran out of inodes a few days ago)
[19:53:34] <jeremyb>	 i *thought* it was fixed
[19:53:46] <facundobatista>	 jeremyb, how can I know which servers (amsterdarm, virginia) I'm hitting? the IP to which nb.wikipedia.org resolved?
[19:54:00] <jeremyb>	 facundobatista: resolve it and then reverse it
[19:54:02] <Clinteger>	 what would a routing issue look like in a packet capture? :S
[19:54:16] <jeremyb>	 Clinteger: errr, idk. you should try mtr
[19:54:20] <jeremyb>	 Clinteger: what OS?
[19:54:25] <Clinteger>	 os x
[19:54:32] <Clinteger>	 what is mtr?
[19:54:35] <jeremyb>	 yeah, mtr if you can get it easily
[19:54:38] <Clinteger>	 ahh
[19:54:42] <jeremyb>	 do you already have macports?
[19:54:44] <Clinteger>	 jeremyb traceroutes seem to be blocked
[19:54:52] <jeremyb>	 errrr, ISP?
[19:54:56] <Clinteger>	 no, but it might be in homebrew
[19:55:01] <Clinteger>	 jeremyb i'm on a university connection
[19:55:03] <facundobatista>	 jeremyb, I'll get this info and come back to you
[19:55:06] <Clinteger>	 "university" it's odd
[19:55:35] <Clinteger>	 but I have problems connecting to bits.wikimedia.org/.../load.php
[19:56:00] <Clinteger>	 so the static resource loader
[19:56:20] <jeremyb>	 facundobatista: well i'm now hunting for the old conversation to refresh my memory. but if you can try to repro from somewhere else (with no proxy, e.g. your laptop) that would be great. or if you have a script that replicates the issue?
[19:56:28] <Clinteger>	 i suspect i might be  having similar problems with other websites, but it's nowhere near as prevalent. with wikimedia sites it happens very very often
[19:56:47] <Clinteger>	 i got a capture in wireshark of an instance where this happened
[19:57:04] <Clinteger>	 it shows many many [TCP Retransmission] GET /en.wikipedia.org/load.php
[19:57:10] <facundobatista>	 jeremyb, I wasn't able to replicate this at all
[19:57:14] <Clinteger>	 but I'm not sure if that's related
[19:57:48] <jeremyb>	 Clinteger: maybe someone else has ideas but if you can't give a traceroute then i think your first step needs to be complaining to the university that wikipedia doesn't work and that you can't diagnose it because you can't traceroute.
[19:58:00] <jeremyb>	 facundobatista: lovely!
[19:58:04] <Clinteger>	 well jeremyb if i refresh the page it usually works
[19:58:08] <facundobatista>	 jeremyb, indeed :/
[19:58:14] <Clinteger>	 and in fact it's not wikipedia, but only the resource loader, lol
[19:58:19] <Clinteger>	 so i just see a lot of css-less wikipedia
[19:58:49] <jeremyb>	 Clinteger: right, but that means wikipedia doesn't work right.
[19:58:56] <Clinteger>	 indeed indeed
[19:58:59] <facundobatista>	 jeremyb, one detail that may help: the other day we artificially increased a lot the queries (testing our own response times) and we didn't get more of this 403s... so it's not matter of hitting some limits...
[20:00:38] <Clinteger>	 jeremyb i will send an email to the network tech support, thanks for listening :)
[20:00:46] <facundobatista>	 jeremyb, and yes, we have an haproxy and a squid between my client and the public internet
[20:01:20] <jeremyb>	 Clinteger: i'd love to hear the outcome
[20:01:31] <Clinteger>	 yes jeremyb it's a very weird problem
[20:01:37] <Clinteger>	 I've noticed with curl I can't reproduce it
[20:01:44] <Clinteger>	 but both with firefox and chrome, this problem happens
[20:01:58] <Clinteger>	 requests to the resource loader just get cut off after a 30s timeout and i can't figure out why
[20:02:02] <jeremyb>	 facundobatista: who is 10.33.16.26 ?
[20:03:32] <facundobatista>	 jeremyb, asking for that info
[20:04:58] <facundobatista>	 jeremyb, that is the haproxy
[20:05:00] <Clinteger>	 jeremyb I think there is something weird going on with the network's connection to the united states. I get ~2.5 MB/sec from european servers, but if it's an american server, my download speed is 50 KB/sec max.
[20:06:53] <jeremyb>	 facundobatista: yeah. so definitely a problem on your end then
[20:07:30] <jeremyb>	 Clinteger: where is the university? are you not consistently hitting the same continent?
[20:07:38] <Clinteger>	 jeremyb I am in France
[20:10:07] <facundobatista>	 jeremyb, fwiw: http://linkode.org/l/oWRovegrFuSCKC7EgaIHS6
[20:10:31] <multichill>	 Than you would be hitting the farm in Amsterdam. Probably your university -> nren of France -> Geant > Wikimedia
[20:11:14] <facundobatista>	 jeremyb, which servers are "esams"?
[20:11:30] <facundobatista>	 multichill, sorry, that was for me?
[20:11:43] <jeremyb>	 facundobatista: not the best test because you could have something in your hosts file. better to just ping -c 1 hostname
[20:11:45] <Clinteger>	 multichill yep http://pasillo.renater.fr/weathermap/weathermap_metropole.html
[20:12:38] <jeremyb>	 facundobatista: anyway, clusters are %(vendor)2s%(nearbyairport)3s . so esams is amsterdam
[20:13:07] <jeremyb>	 Clinteger: you made it sound like you were sometimes hitting amsterdam and sometimes virginia?
[20:13:10] <jeremyb>	 as i understood it
[20:13:15] <Clinteger>	 jeremyb wut
[20:13:22] <Clinteger>	 oh jeremyb no no, there is only one bits.wikimedia.org
[20:13:32] <jeremyb>	 how do you figure?
[20:13:37] <Clinteger>	 O_o is there more than one?
[20:14:06] <jeremyb>	 hrmmm, idk what the current configuration is though
[20:14:17] <jeremyb>	 i just queried from france and got eqiad
[20:14:56] <Clinteger>	 eqiad?
[20:16:00] <jeremyb>	 Clinteger: 14 20:12:38 < jeremyb> facundobatista: anyway, clusters are %(vendor)2s%(nearbyairport)3s . so esams is amsterdam
[20:16:04] <jeremyb>	 eqiad is virginia
[20:16:08] <Clinteger>	 oh shucks
[20:16:20] <Clinteger>	 bits-lb.esams.wikimedia.org
[20:16:21] <Clinteger>	 that's what i get
[20:16:39] <Clinteger>	 hmm, well that throws my idea out the window
[20:16:43] <jeremyb>	 oh, ok. then that's europe
[20:17:05] <Clinteger>	 hmm, sorry, i wasn't aware
[20:17:20] <jeremyb>	 anyway, i'm a little confused about what your symptoms are but i still think the next step is talk to the uni
[20:18:14] <Clinteger>	 unfortunately in france, august is the month of vacations, so there is hardly anyone around
[20:18:18] <facundobatista>	 jeremyb, help me to understand this better: why you say that the problem is in my end? reading the error body, it seems that the squid is saying that it got an "Access denied" when trying to hit "http://nb.wikipedia.org/w/api.php?"
[20:18:57] <facundobatista>	 jeremyb, the squid is the last node in the border of the public internet, so the "Access denied" seems to come from outside
[20:19:20] <jeremyb>	 facundobatista: that message is generated by your squid. not ours. you need to figure out exactly what the problem is. squid overloaded? squid got connection refused?
[20:20:13] <facundobatista>	 jeremyb, squid got Access denied, it's what the body says
[20:20:14] <jeremyb>	 facundobatista: you don't control the squid yourself i guess
[20:20:23] <jeremyb>	 facundobatista: what does that mean though?
[20:20:34] <jeremyb>	 squid got a 403?
[20:20:48] <jeremyb>	 why is it not passing through the 403 then?
[20:21:26] <Clinteger>	 http://i.imgur.com/NLxheGA.png this is what traceroute looks like
[20:21:47] <Clinteger>	 this traceroute is actually from an upstream router in paris
[20:21:50] <jeremyb>	 you have an internal (RFC1918) IP that exists on your network and is missing from our rdns so probably doesn't exist on our network mentioned in the body. so that body was generated by you not us
[20:22:23] <jeremyb>	 Clinteger: :(
[20:23:01] * jeremyb  waits for ^d to finish syncing :)

[20:23:03] <facundobatista>	 jeremyb, yes, that body is generated from my squid, it seems it got an "Access denied", but don't know what that means either
[20:23:09] <Clinteger>	 people in ##networking told me to try tcptraceroute, all the protocol options in standard bsd traceroute…none of them work, heh
[20:23:17] <jeremyb>	 facundobatista: well find out. tcpdump if you have to :)
[20:23:21] <facundobatista>	 jeremyb, I'll try to pursue this internally
[20:23:37] <facundobatista>	 jeremyb, will come back with further news, thank you very much for your help!!
[20:24:50] <jeremyb>	 facundobatista: sure
[20:30:53] <jeremyb>	 > Issues with Wikimedia's installation of Git/Gerrit. Note that repository creation requests should be filed here instead of Bugzilla.
[20:31:01] <jeremyb>	 ^d: should that mention gitblit?
[20:31:31] <jeremyb>	 that's a lot of default cc :)
[20:31:47] <^d>	 jeremyb: Yeah, that's the right place. I'll tweak the description.
[20:40:52] <jeremyb>	 i'm surprised there's so many proxies in google's index... (and not automatically deduped!)
[20:40:58] <jeremyb>	 https://www.google.com/search?q="For+contributing+to+Gerrit%2C+you+need"
[22:08:48] * Romaine  points at https://bugzilla.wikimedia.org/show_bug.cgi?id=52853

[22:19:11] <jeremyb>	 Romaine: that's a fun one!
[22:19:13] * jeremyb  runs awya

[22:19:15] <jeremyb>	 away*
[22:19:20] <Romaine>	 :p :p
[22:19:57] <Romaine>	 I think that is the corpse in the closet of the software ;) finally found!
[22:20:40] <Romaine>	 "Lijk in de kast" is Dutch expression
[22:21:00] <Romaine>	 I mean skeleton in the closet
[22:23:51] <marktraceur>	 *the*? Haven't we found more?
[22:24:30] <marktraceur>	 In fact, 52852 before that one
[22:24:55] <Romaine>	 mmm, I never read about that in the Signpost
[23:24:51] <DaBPunkt>	 gn8 folks