[00:00:29] <James_F>	 We fixed it within 6 hours, but MapBox didn't and it bit a lot of their users: https://nymag.com/intelligencer/2018/08/why-was-nyc-renamed-jewtropolis-on-snapchat.html
[00:01:06] <AntiComposite>	 that's what happens when you only have fancy "ai" systems determining if changes should get through
[00:01:33] <James_F>	 They have users doing it, but unfortunately not enough people looking at their "recent changes" feed.
[00:01:40] <James_F>	 Which is a situation I'm sure we can empathise with. :-(
[00:01:48] * AntiComposite edits OSM as well
[00:14:44] <Platonides>	 and it's probably not easy at all to validate geo changes if you don't know the zone
[00:18:26] <AntiComposite>	 If I remember correctly, mapbox uses two methods to determine if a changeset (edit) is harmful: one is stats and AI, the other is time. The more the stats/AI say the changeset is harmful, the longer it will have to stay before Mapbox will include it
[00:49:12] <cdanis>	 AntiComposite: there's not been a policy decision about whether or not to 'fully' block external usage of maps.wm.o -- right now it will still work if you happen to request a maptile that is in the CDN cache (which happens to be a lot of them)
[00:50:15] <AntiComposite>	 Yeah, I told them that it's currently disabled for an unknown period of time and that a different tileserver would be a good idea
[00:50:43] <cdanis>	 perfect, thanks :)
[01:10:28] <Nemo_bis>	 "Funny", just the other week I sent https://lists.wikimedia.org/pipermail/maps-l/2020-January/001704.html
[01:12:01] <Nemo_bis>	 Because I think that even without a clearcut policy it's helpful to come up with a couple examples of obviously ok and obviously bad usage
[14:01:35] <robbi5>	 hey, i'm currently getting more and more http 429 from maps.wikimedia.org osm-intl tiles. i haven't found matching phabicator tasks yet, but the daily-tile-requests graph on the interactive-team-kpi dashboard in grafana seems to show also a large decrease in requests. was a new policy or config change introduced recently?
[16:16:43] <cdanis>	 robbi5: not a new policy, although we're considering one.  Maps has had several outages recently where one of the causes is a chronic underprovisioning of the service -- the servers are often running in excess of 75% cpu utilization and sometimes pegged at 100%.  we've temporarily started returning HTTP 429 for non-Wikimedia-wiki users in the event that the tiles they're requesting aren't already in
[16:16:45] <cdanis>	 the CDN cache, as an outage mitigation strategy
[16:17:16] <cdanis>	 I wish I could tell you for how long that will be in place, but I don't know that
[16:35:42] <robbi5>	 cdanis: oh. okay, thank you :) we've using tiles with a high zoom level, so thats why the tiles aren't cached and we get the 429. can you add this note somewhere on a wikipage of the map service? i've seen people with the same question even in the software repo: https://github.com/kartotherian/kartotherian/issues/121 :)
[17:02:51] <cdanis>	 robbi5: ah, good to know, thanks.  I'll at least update the Phab task from the outage with details about the current situation, and communicate that around
[17:03:47] <RhinosF1>	 Does any know an alternative to wgHostname that gives the data centre as well?
[17:04:10] <RhinosF1>	 I have the code in https://meta.wikimedia.org/wiki/User:RhinosF1/global.js
[17:05:23] <RhinosF1>	 Which returns 199ms (mw1319) but I want something like 199ms (mw1319.server.wikimedia.org)
[17:05:38] <RhinosF1>	 Where server is equiad, essams etc
[17:07:19] <Lucas_WMDE>	 not sure if that exists, but I suppose something like
[17:07:21] <Lucas_WMDE>	 ( { 1: 'eqiad', 2: 'codfw', 3: 'esams', 4: 'ulsfo', 5: 'eqsin' } )[ mw.config.get( 'wgHostname' ).match(/(\d)\d{3}/)[1] ]
[17:07:23] <Lucas_WMDE>	 would work
[17:07:28] <Lucas_WMDE>	 (list based on https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Servers)
[17:07:30] <cdanis>	 that's the hostname of the mediawiki server which means two things, 1) the FQDN will be something like 'mw1319.eqiad.wmnet' which isn't a real TLD, 2) likely it will only ever be eqiad or codfw, 3) mw1* will always be eqiad and mw2* will always be codfw
[17:08:12] <cdanis>	 I'd guess that we don't expose the '.eqiad.wmnet' portion because it isn't meaningful outside WMF private networks, and also doesn't add information
[17:08:59] <RhinosF1>	 cdanis: then I can probably just note it and remember. I just wanted to be able to tell which centre it was serving it
[17:10:24] <cdanis>	 yeah, what Lucas said will work well for you
[17:10:41] <cdanis>	 this of course doesn't tell you anything about the CDN servers involved, but that's in the X-Cache response header if you need it
[17:11:04] <Lucas_WMDE>	 yeah, I’m basically exploiting 3) of what cdanis said
[17:11:16] <Lucas_WMDE>	 that the first digit of the server name gives you the dc
[17:13:18] <_joe_>	 RhinosF1: https://config-master.wikimedia.org/discovery/discovery-basic.yaml tells you which datacenters are active currently
[17:13:50] <_joe_>	 the - dnsdisc: "/appservers-rw"
[17:14:08] <_joe_>	 and - dnsdisc: "/api-rw" are the records you're interested into
[17:14:26] <_joe_>	 will tell you which datacenter is serving mediawiki traffic right now
[17:15:12] <_joe_>	 if you want to know which applayer is responding you
[17:15:39] <RhinosF1>	 Thanks _joe_
[17:15:46] <_joe_>	 for what you want, though, the easiest way to get that info is
[17:15:51] <_joe_>	 mw1* => eqiad
[17:15:56] <_joe_>	 mw2* => codfw
[17:17:32] <RhinosF1>	 What Lucas_WMDE gave works
[17:17:53] <_joe_>	 yep
[20:25:37] <Nemo_bis>	 Commons doesn't like me today
[20:25:38] <Nemo_bis>	 Request from - via cp3064.esams.wmnet, ATS/8.0.5 Error: 502, Next Hop Connection Failed at 2020-02-06 20:24:52 GMT
[20:25:54] <Nemo_bis>	 And even a good old bare: 504 Gateway Time-out nginx/1.13.9
[20:26:05] <Praxidicae>	 uh
[20:26:08] <Praxidicae>	 did wp die again
[20:26:15] <guillom>	 Nemo_bis: yeah, not just you
[20:26:21] <guillom>	 It's not personal :)
[20:26:27] <Nemo_bis>	 I'm glad ;) 
[20:26:46] <Praxidicae>	 #rip
[20:26:59] <paladox|UKInEU>	 gone down for me too
[20:28:14] <Nemo_bis>	 (ops called in en masse; revert upcoming)
[20:29:19] <Praxidicae>	 who b0rked it 
[20:30:02] <rlazarus>	 thanks for the reports, team is already working on it <3
[20:30:08] <DSquirrelGM>	 I'm getting major issues trying to get the upload forms to try to load on Commons right now, too
[20:30:50] <guillom>	 It'll take a little time for the revert to go through.
[20:31:22] <rr0>	 Nothing works now
[20:31:32] <Praxidicae>	 :/
[20:31:35] <Praxidicae>	 yah this is pretty bad 
[20:32:17] <iridescence>	 en.wikipedia works kind of
[20:32:38] <paladox|UKInEU>	 works for me again
[20:34:01] <Nemo_bis>	 looks like we're back
[20:34:36] <RhinosF1>	 Meta was just slow actually trying to load the page with my issue on but I broke paladox|UKInEU’s response time script
[20:34:49] <rlazarus>	 everything should be recovered, let us know if you're still seeing issues
[20:35:04] <RhinosF1>	 I’m getting ERRORJavaScript parse error: Parse error: Missing ; before statement in file 'User:RhinosF1/global.js' on line 13 on
[20:35:05] <paladox|UKInEU>	 script?
[20:35:06] <RhinosF1>	 https://meta.wikimedia.org/w/index.php?title=User:RhinosF1/global.js&diff=19779560&oldid=19779344
[20:35:18] <RhinosF1>	 paladox|UKInEU: from your mh global.js
[20:36:04] <paladox|UKInEU>	 it's not mine, it's SPF|Cloud 
[20:36:46] <RhinosF1>	 paladox|UKInEU: fair enough
[20:40:50] <RhinosF1>	 paladox|UKInEU: any idea how to fix it?
[20:42:51] <paladox|UKInEU>	 maybe a missing comma
[20:43:11] <RhinosF1>	 paladox|UKInEU: where exactly?
[20:43:25] <paladox|UKInEU>	 dcas = ( { 1: 'eqiad', 2: 'codfw', 3: 'esams', 4: 'ulsfo', 5: 'eqsin' } )[ mw.config.get( 'wgHostname' ).match(/(\d)\d{3}/)[1] ] -> dcas = ( { 1: 'eqiad', 2: 'codfw', 3: 'esams', 4: 'ulsfo', 5: 'eqsin' } )[ mw.config.get( 'wgHostname' ).match(/(\d)\d{3}/)[1] ],
[20:44:30] <RhinosF1>	 Nope paladox|UKInEU
[20:44:57] <RhinosF1>	 https://meta.wikimedia.org/w/index.php?title=User:RhinosF1/global.js&diff=19780060&oldid=19779560 gets the same error
[20:45:38] <paladox|UKInEU>	 maybe caption  = respTime.toString() + 'ms (' + server + '.' + dcas ')'; -> caption  = respTime.toString() + 'ms (' + server + '.' + dcas + ')';
[20:48:06] <RhinosF1>	 paladox|UKInEU: yep thx
[21:59:38] <DSquirrelGM>	 got tired of getting randomly disconnected due to "invalid captcha" so I switched to a standalone IRC client instead of webchat