[06:31:30] Hi guys, I was trying to retrieve my logins on the website but I forgot which email it was. I joined but never had a chance to login. Platonides mentioned about requesting to usurp the username. My question is should I create a new account because based on the Instructions "Log in to your present account."? [12:34:37] any hint where can I found parser profiling information? [12:34:55] They seem to disappear from submit page :( [17:45:05] ping paravoid [18:44:01] travelling I think, Platonides [20:37:12] I am creating a gadget which has a primary button at the toolbar. It will have three options that will show if the button is clicked. I am wondering how can be the option buttons created. [20:56:53] Does anybody know why https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Fossilised_frog.jpg/800px-Fossilised_frog.jpg is serving a zero-length body? [20:56:56] At least for me it is [20:57:41] $ curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Fossilised_frog.jpg/800px-Fossilised_frog.jpg [20:57:41] Content-Type: image/jpeg [20:57:41] Content-Length: 0 [21:04:34] Jasper, uh oh... this again [21:04:39] bblack, ^ [21:04:45] alex@alex-laptop:~$ curl -Ik -H "Host: upload.wikimedia.org" https://upload-lb.{eqiad,codfw,esams,ulsfo}.wikimedia.org/wikipedia/commons/thumb/1/17/Fossilised_frog.jpg/800px-Fossilised_frog.jpg 2>/dev/null | grep Content-Length: [21:04:45] Content-Length: 160047 [21:04:45] Content-Length: 160047 [21:04:45] Content-Length: 160047 [21:04:45] Content-Length: 0 [21:05:17] This was https://phabricator.wikimedia.org/T144257 [21:06:08] content-length:160047 [21:06:23] I'm running it against the eqiad lb [21:06:25] Krenair, thanks for looking into it for us. [21:06:34] We pretty regularly scrape these things and find them quite fast :) [21:07:07] Krenair, is there a way of knowing it came from ulsfo btw? [21:07:16] I don't see it in the response headers. [21:07:26] yes [21:07:28] X-Cache [21:07:35] gives you the hostnames involved [21:07:45] I got "X-Cache: cp1050 miss, cp2002 miss, cp4015 hit/8, cp4006 miss" when I tried ulsfo [21:07:46] ah ulsfo is the 0 returner [21:07:47] I see [21:07:47] X-Cache: cp1050 miss, cp2002 miss, cp4015 hit/5, cp4014 hit/3 [21:08:00] all cache box hostnames begin with cp [21:08:01] Right, so I assume you have some internal map for this. [21:08:06] the first number is the DC number [21:08:12] that e.g. cp4014 is ulsfo [21:08:13] 1 for eqiad, 2 for codfw, 3 for esams and 4 for ulsfo [21:09:20] https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Servers [21:10:10] Cool, so I'll just sit back and let you guys figure it out. [21:10:27] Sometimes we do see that it returns application/x-www-urlencoded as the Content-Type [21:10:36] pmtpa/sdtpa is listed there though we no longer have any servers there, as of some years ago [21:10:38] well [21:10:41] I've done all I can [21:10:47] If you need any other help / info, I'll let you know. [21:10:49] OK. [21:10:53] probably [21:11:25] It may be a bad idea to purge anything before getting traffic ops to look at it [21:11:26] er, that was supposed to say "you let me know" [21:15:30] apergos, I guess there's no easy way to tell how many objects ulsfo servers have cached as content-length 0? [21:16:11] I have no idea, that's a bblack question [21:16:42] thought so [21:17:01] thing is neither him nor ema are around and I don't know what the scale of this problem is [21:17:08] did you find this with any other images Jasper? [21:21:02] https://wikitech.wikimedia.org/wiki/Varnish#One-off_purges_.28bans.29 [21:21:15] we could consider this but I would really really rather wait for him or ema [21:21:40] yes agreed [21:21:44] well especially I would rather not be here to baby sit it, as my brain has mostly clocked out already (midnight:30) [21:22:02] but note the example :-P [21:22:54] oh right it was literally written for this bug last time it surfaced [21:26:03] but [21:26:06] https://gerrit.wikimedia.org/r/#/c/307742/ [21:26:17] so if we don't even cache those... [21:26:22] ?! [21:28:04] bblack is in the sf tz and was around earlier, so I would say let's punt to him at this point (if he doesn't show up soonish, sms?) [21:28:50] irc idle time is an hour and a half [21:29:56] https://phabricator.wikimedia.org/T131502 see also this [21:30:07] upgraded today (monday) [21:31:15] Krenair, we've found some before -- it seems to be pretty rare. [21:31:34] Krenair, we're actually more used to valid data but with a bogus content-type [21:31:45] before = today? [21:31:48] who is we? [21:31:56] or do you mean a week or more ago, or...? [21:32:06] if I may ask [21:32:38] The last time I saw a Content-Length: 0 image must have been 4-5 months ago. [21:32:56] As for Content-Type: application/x-www-urlencoded, we see it a lot more often. [21:33:28] We wrote code to just autodetect Content-Type based on the extension in the URL. [21:33:50] "We" = the team at Endless. If you see openendless@endlessm.com as a User-Agent in your logs, that's us [21:34:31] ok, so no more today. hm [21:34:58] Not so far. [21:35:12] we will definitely want to hear about it if you come across more [21:35:15] I found this one doing routine development on a project here. [21:35:25] Will do. [21:36:29] Platonides: what's up? [21:40:05] hi paravoid [21:40:23] hi [21:40:29] saw you as a maintainer for a geoip mmdb package in debian [21:40:46] wondered if you knew of the equivalent to geoip-database{,-extra,-contrib} for geoip2 [21:40:54] there isn't one yet [21:41:18] I want to ync up with the geoip-database maintainer at some point, haven't found the time :( [21:41:23] sync* [21:41:56] ☹ [21:42:40] seemed odd that there wasn't one [21:43:49] I think I'll guess where it would go, then [21:43:59] /usr/share/GeoIP/ perhaps? [21:44:10] probably [21:47:44] well, I look forward to seeing such package ;) [21:49:31] these packages are a bit tricky, since they need to be kept updated [22:02:02] Jasper: right now it's unreproduceable for us [22:03:00] (as in, I can poll that image URL on all cache frontends, and they all return the right CL) [22:03:51] but also, there was a known-issue window around 20:04 -> 21:00 UTC, so I wouldn't be surprised at seeing it during that [22:04:08] it looks like Krenair's test was right near the end of that [22:04:17] and the report came during it [22:25:58] bblack: I still get CL:0 for [22:26:00] curl -Ik -H "Host: upload.wikimedia.org" https://upload-lb.ulsfo.wikimedia.org/wikipedia/commons/thumb/1/17/Fossilised_frog.jpg/800px-Fossilised_frog.jpg 2>/dev/null | grep Content-Length: [22:26:32] same for https://upload.wikimedia.org/wikipedia/en/thumb/9/9e/1806flag.jpg/220px-1806flag.jpg [22:27:12] tgr: I still can't reproduce [22:27:26] and I'm hitting all of the caches [22:28:12] also, can you move this to #wikimedia-traffic ? it's better to keep the conversation in one place [22:28:34] sorry, wasn't aware its discussed there [22:42:01] bblack, still seeing CL:0 as well [22:42:13] This is from curl which I doubt caches anything. [22:42:21] oh moving it to #wikimedia-trafic