[00:08:24] <Krinkle>	 Reedy: Hm.. I suppose I could request to *-lb.pmtpa.wikimedia.org/w/api.php to get replag (and eqiad similarly) and forge the Host header to lang.site.org (so that it serves MediaWiki instead of www.wikimedia.org)
[00:10:06] <Krinkle>	 looks like cURL supports this
[00:10:07] <Krinkle>	 http://drewish.com/content/2010/03/using_curl_and_the_host_header_to_bypass_a_load_balancer
[00:10:10] <Krinkle>	 lets see
[00:14:46] <Krinkle>	 Reedy: Interesting, it does work but I get the same db-hosts for both (including the slaves)
[00:15:07] <Krinkle>	 Though cache handling does look very different
[00:15:21] <Krinkle>	 pmtpa:
[00:15:23] <Krinkle>	 < X-Cache: MISS from sq36.wikimedia.org
[00:15:23] <Krinkle>	 < X-Cache-Lookup: MISS from sq36.wikimedia.org:3128
[00:15:23] <Krinkle>	 < X-Cache: MISS from sq66.wikimedia.org
[00:15:25] <Krinkle>	 < X-Cache-Lookup: MISS from sq66.wikimedia.org:80
[00:15:32] <Krinkle>	 eqiad:
[00:15:35] <Krinkle>	 < X-Cache: MISS from cp1005.eqiad.wmnet
[00:15:35] <Krinkle>	 < X-Cache-Lookup: MISS from cp1005.eqiad.wmnet:3128
[00:15:36] <Krinkle>	 < X-Cache: MISS from cp1001.eqiad.wmnet
[00:15:38] <Krinkle>	 < X-Cache-Lookup: MISS from cp1001.eqiad.wmnet:80
[00:31:47] <Reedy>	 Eh?
[00:31:53] <Reedy>	 Backend and frontend squids
[00:55:18] <Krinkle>	 @replag all
[00:55:20] <dbbot-wm>	 Krinkle: [s1] db1017: 0s, db1001: 0s, db1042: 0s, db1049: 0s, db1050: 0s, db1043: 0s; [s2] db1034: 0s, db1002: 0s, db1009: 0s, db1018: 0s; [s3] db1019: 0s, db1003: 0s, db1010: 0s, db1035: 0s
[00:55:21] <dbbot-wm>	 Krinkle: [s4] db1038: 0s, db1004: 0s, db1011: 0s, db1020: 0s; [s5] db1039: 0s, db1005: 0s, db1026: 0s, db1021: 0s; [s6] db1006: 0s, db1022: 0s, db1027: 0s, db1040: 0s; [s7] db1041: 0s, db1007: 0s, db1024: 0s, db1028: 0s
[00:55:29] <Krinkle>	 Reedy: Okay, now it works. Set to eqiad for now.
[00:56:06] <Krinkle>	 Reedy: Is it on purpose that the lb for pmtpa responds from eqiad? I guess the primary is set on a deeper level
[00:56:36] <Krinkle>	 I figured pointing to its IP directly would bypass the database picker
[00:58:00] <Krinkle>	 curl --verbose --header 'Host: en.wikipedia.org' 'wikipedia-lb.eqiad.wikimedia.org:80/w/api.php?format=json&action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb=1'
[00:58:21] <Krinkle>	 curl --verbose --header 'Host: en.wikipedia.org' 'wikipedia-lb.pmtpa.wikimedia.org:80/w/api.php?format=json&action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb=1'
[01:20:30] <Krinkle>	 Reedy: Hm.. I'm getting 403 error from bits on nl.wikipedia.org
[01:20:33] <Krinkle>	 https://bits.wikimedia.org/nl.wikipedia.org/load.php?debug=false&lang=nl&modules=ext.gadget.Direct-link-to-Commons%2CIProject%2COpenStreetMapFrame%2CProtectionTemplates%7Cext.wikihiero%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmw.PopUpMediaTransform%7Cskins.vector&only=styles&skin=vector&*
[01:20:35] <Krinkle>	 Scripts should use an informative User-Agent string with contact information, or they may be IP-blocked without notice.
[01:20:38] <Krinkle>	 </quote>
[01:20:43] <Krinkle>	 Cache fuck up?
[01:21:08] <Krinkle>	 weird
[01:21:12] <Krinkle>	 Now it works
[01:21:47] <Krinkle>	 I was able to reproduce it on cold cache
[01:22:36] <Krinkle>	 Getting it again now
[03:22:53] <Reedy>	 Nemo_bis: Done! :p
[04:06:20] <Nemo_bis>	 ou
[04:10:35] <Nemo_bis>	 Wantedpages seems quite harmless in the end
[16:07:41] <Reedy>	 Nemo_bis: https://noc.wikimedia.org/~reedy/enwikispecialpages.log
[20:22:50] <Nemo_bis>	 db60 complains a bit but nothing tragic
[21:00:10] <jeremyb>	 Nemo_bis: where?
[21:01:23] <Amgine>	 <extremely innocent look> JeremyB: the internet.
[21:07:30] <jeremyb>	 Amgine: i'm really going to rip out that shift key! :-)
[21:08:37] <Nemo_bis>	 jeremyb: https://noc.wikimedia.org/dbtree/
[21:08:44] <Nemo_bis>	 (now it's fine)
[21:10:00] <jeremyb>	 oh
[21:11:03] <Amgine>	 <gg>
[22:36:43] <twood>	 Is there a reason why the wikimedia ganglia site was recently password protected?  I'm a Comp Sci prof that was hoping to use some of that data for my research.
[22:37:56] <paravoid>	 twood: some ganglia vulnerabilities came to our attention
[22:38:10] <paravoid>	 it's temporary, until they get fixed
[22:39:51] <twood>	 ah, ok.  Any estimate if that is days/weeks/months away?
[22:40:12] <paravoid>	 hopefully days/weeks
[22:40:42] <paravoid>	 were you thinking of a particular graph?
[22:40:44] <paravoid>	 I could get it for you
[22:43:09] <twood>	 Not exactly sure at the moment--we had been looking at the memory data across machines.  We can wait for now.  It's one of the few public sources of info on a *real* website. We appreciate that you guys share it
[22:43:30] <paravoid>	 the intention is to keep sharing it
[22:43:54] <paravoid>	 but in the face of a real vulnerability, we had no other option
[22:44:11] <paravoid>	 s/real/known/
[22:47:25] <twood>	 yep, makes sense.  I'll check back in a few weeks.  thanks for the info!
[23:32:05] <DaBPunkt>	 gn8 folks