[00:00:48] * RoanKattouw blames Krinkle , partially [00:03:32] Krinkle: https://gerrit.wikimedia.org/r/#/c/94708/ [00:09:00] ori-l: Timo knows what's going on and is fixing [00:09:16] cool, thanks [00:12:22] MaxSem: Are you doing your LD now? [00:12:46] RoanKattouw: fix https://gerrit.wikimedia.org/r/#/c/74400/ I command you [00:13:05] Oh right that one [00:13:07] sigh [00:13:14] thou cluttereth thine dashboard [00:13:15] poor Psi IM wiki http://psi-im.org/wiki/ [00:13:47] (03PS1) 10Chad: Fix test2wiki to only have Cirrus as primary, not both [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99007 [00:13:54] RoanKattouw, awwww - totally forgot:( [00:13:58] (03CR) 10Chad: [C: 032 V: 032] Fix test2wiki to only have Cirrus as primary, not both [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99007 (owner: 10Chad) [00:14:00] feel free to deploy [00:14:10] I'll be ready after that [00:14:14] sorry [00:14:32] OK [00:14:34] Going now [00:14:50] !log demon synchronized cirrus.dblist 'Fix test2wiki to have Cirrus as primary' [00:14:53] mutante: we heard you like exceptions so we put an exception inside your exception, etc. [00:15:01] ^d: Dude? [00:15:06] Logged the message, Master [00:15:08] Scheduled deploy window? [00:15:27] <^d> LD. [00:15:37] Yes [00:15:47] ori-l: haha, yea, exceptions in the exception handler are the best exceptions? [00:15:53] Which you didn't list yourself for [00:15:59] And you didn't communicate with the people that had listed themselves [00:16:35] <^d> Sorry the loop didn't get closed. It was discussed and approved. [00:16:39] OK [00:16:41] Are you done? [00:16:41] <^d> It should've gotten on-wiki. [00:16:43] <^d> Yes. [00:16:47] Cause then I'll go next, and Max after me [00:16:51] Sweet [00:17:06] (03CR) 10Catrope: [C: 032] Disable VisualEditor in content namespaces on svwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98871 (owner: 10Catrope) [00:17:35] mutante: :P [00:18:23] (03Merged) 10jenkins-bot: Disable VisualEditor in content namespaces on svwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98871 (owner: 10Catrope) [00:19:58] !log catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgVisualEditorInContentNamespaces and set it to false on svwiktionary' [00:20:14] Logged the message, Master [00:21:00] !log catrope synchronized wmf-config/CommonSettings.php 'Add plumbing for wmgVisualEditorInContentNamespaces' [00:21:14] Logged the message, Master [00:30:38] !log catrope synchronized php-1.23wmf4/extensions/VisualEditor/VisualEditor.php 'Blacklist IE11' [00:30:52] Logged the message, Master [00:30:53] !log catrope synchronized php-1.23wmf5/extensions/VisualEditor/VisualEditor.php 'Blacklist IE11' [00:30:57] yurik: https://gerrit.wikimedia.org/r/#/c/97130/ is still unmerged [00:31:10] (03Abandoned) 10Chad: Make Cirrus the default on test2wiki again [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98901 (owner: 10Manybubbles) [00:31:59] paravoid, darn, totally missed it - if you can +2 it, will push it out right now (unless anyone is deploying?) [00:32:05] greg-g, lightning? [00:32:06] yurik: also, X-Subdomain is wrong there, there's not going to be an X-Subdomain [00:32:19] paravoid, why not? [00:32:21] so fix that too [00:32:40] paravoid, are you saying it won't go through the varnish zero detection? [00:32:46] because than it won't work [00:33:17] we need it to go through zero netmapper to figure out who it is [00:33:27] hm, I guess it will have it set [00:33:53] but why do you need it anyway? Host is sufficient for that, isn't it? [00:34:14] Logged the message, Master [00:34:25] i.e. it will never be Host: m.wikipedia.org X-Subdomain: ZERO or Host: zero.wikipedia.org X-Subdomain: M [00:34:30] paravoid, absolutelly not - some carriers allow one without the other [00:34:37] oh, that [00:35:04] paravoid, https://gerrit.wikimedia.org/r/#/c/98998/ [00:35:17] that patch looks at the HOST if X-SUBDOMAIN is not set [00:35:51] so if you want, X-SUBDOMAIN doesn't have to be really set for the redirector after that patch goes out. [00:36:02] but for now, i am guessing it will be set anyway? [00:36:32] it will be, but the Vary is redundant [00:37:09] paravoid, wouldn't varnish remove m. and zero. in front? [00:37:13] Varnish will set X-Subdomain, you don't need the Vary (but it won't hurt) [00:37:14] if it won't, than yes [00:37:32] it won't [00:37:43] it currently does, but i will need to check the regex it uses [00:37:43] if it did, the virtualhost wouldn't work anyway [00:37:52] good point [00:37:58] ok, i can remove taht [00:38:09] I mean, it won't change anything in practice [00:38:23] but since we haven't merged that "fix Vary" patchset... :-) [00:39:48] I'm okay if you want to leave it there [00:40:04] RoanKattouw, are you done? [00:40:04] (03PS2) 10Yurik: Mobile redirect - changed cache Vary header [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97130 [00:40:13] paravoid, ^ [00:40:15] MaxSem: Sorry [00:40:17] yes [00:40:19] :) [00:40:24] I was done then immediately went to the restroom, sorry :S [00:40:26] (03CR) 10MaxSem: [C: 032] Fix removal of title coordinates from extracts [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98848 (owner: 10MaxSem) [00:40:35] (03Merged) 10jenkins-bot: Fix removal of title coordinates from extracts [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98848 (owner: 10MaxSem) [00:40:44] MaxSem, are you deploying? [00:40:51] yep [00:40:54] (03CR) 10Faidon Liambotis: [C: 032] Mobile redirect - changed cache Vary header [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97130 (owner: 10Yurik) [00:41:01] MaxSem, could you grab https://gerrit.wikimedia.org/r/#/c/97130/ as well? [00:41:02] (03Merged) 10jenkins-bot: Mobile redirect - changed cache Vary header [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97130 (owner: 10Yurik) [00:41:11] its a tiny vary header change [00:42:09] sure [00:43:20] !log maxsem synchronized wmf-config/InitialiseSettings.php [00:43:25] yurik: funny how it redirects on everything you throw at it as a request URL [00:43:34] Logged the message, Master [00:43:40] GET /favicon.ico HTTP/1.1 [00:43:44] HTTP/1.1 302 Found [00:43:46] Location: https://en.m.wikipedia.org/wiki/Main_Page [00:44:33] anyway [00:44:35] off for the night [00:44:38] paravoid, heh, its a redirector: ) But yes, favicon should probably not be a redirect..? [00:44:47] !log maxsem synchronized mobilelanding.php [00:44:51] in reality noone should ever hit favicon [00:44:59] thx MaxSem ! [00:45:03] Logged the message, Master [00:46:13] yurik: is the mobile team aware of the m.wp.org change? [00:46:31] the "mobile web" team I guess we call it now ;-) [00:47:12] i haven't told them explicitly, but we are not changing the behaviour [00:47:23] because that's the same as hardcoded in varnish [00:47:32] hm, I guess that's right [00:48:06] right, nevermind [00:48:39] we're actually fixing the https case too :) [00:48:54] if you go to https://m.wp.org now it redirects to http [00:49:04] not that anyone would do that I guess [00:49:27] Krinkle: do you have a time-frame for a fix to the ResourceLoaderLanguageDataModule mtime issue? [00:49:38] ori-l: I do [00:49:41] ori-l: negative 5 minutes [00:49:48] lol [00:49:53] he's good [00:50:36] In style of Bill and Ted: Don't forget to actually go back in time once you get the time machine. but considering it is in gerrit, I guess that's proof we'll eventually have time travel :) [00:50:37] i prefer to use "i" [00:51:06] (the car keys behind the bushes in the Excellent Adventure, for those who know the movie) [00:51:07] (noncapitalized) [00:51:31] paravoid, the vary header has been pushed out btw [00:51:34] ori-l: https://gerrit.wikimedia.org/r/#/c/99010/ [00:51:36] thx to max [00:51:41] yeah, looking it over [00:51:47] yurik: cool [00:52:10] yurik: do you see any reason for us to not make the rewriterule ^/$ ? [00:52:19] not really [00:52:25] its a bouncer, noone should ever stay on it [00:53:06] i mean what else will we return - 404? [00:53:10] yes [00:53:19] RoanKattouw: Krinkle's change looks good to me; do you want to have a look before I merge? [00:53:26] Looking [00:53:45] My commit message to diff ratio will rise if this gets merged :) [00:53:49] yurik: also note (and review) https://gerrit.wikimedia.org/r/#/c/98058/ so that we can get rid of the other varnish redirect too [00:53:53] and now I'm off [00:54:07] * ori-l waves [00:54:56] s/langauge/language/ [00:54:58] on that commit msg [00:54:59] paravoid, goodnight, and that would be a separate submit [00:55:10] we would have to get biz ppl involved [00:55:13] yurik: yes it will be [00:55:21] why? it doesn't change anything [00:55:34] no idea who uses or why we have that, or if we have contracts for that [00:55:46] go sleep already [00:56:34] that change doesn't change any business logic [00:56:51] the only functional change is breaking the old ruby gateway URLs but I don't think we should care anymore [00:57:56] Krinkle: cherry-picks? [01:02:03] I'm not deploying right now [01:02:31] Unless we can get a window, in which I'll be glad to take it all the way [01:02:37] case* [01:02:43] i asked greg-g in -dev [01:02:50] ..and he just said yes [01:02:53] and I said yes [01:02:58] ok [01:03:07] ori-l: go ahead then :) [01:03:07] want me to do it? [01:03:09] k [01:03:12] sure [01:07:28] (03PS2) 10Tim Starling: Add "touch.php" for $wgAppleTouchIcon... [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90886 (owner: 10Reedy) [01:07:48] (03CR) 10Tim Starling: [C: 032] Add "touch.php" for $wgAppleTouchIcon... [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90886 (owner: 10Reedy) [01:08:38] !log ori synchronized php-1.23wmf4/includes/resourceloader/ResourceLoaderModule.php 'Ifa9088c11: resourceloader: Make sure hashmtime cache key is different by language' [01:09:38] Logged the message, Master [01:10:43] !log ori synchronized php-1.23wmf5/includes/resourceloader/ResourceLoaderModule.php 'Ifa9088c11: resourceloader: Make sure hashmtime cache key is different by language' [01:10:58] Logged the message, Master [01:13:50] (03Merged) 10jenkins-bot: Add "touch.php" for $wgAppleTouchIcon... [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90886 (owner: 10Reedy) [01:15:43] Krinkle: module version looks stable now [01:18:09] (03CR) 10Tim Starling: [C: 04-1] "The apache config looks OK already, but it definitely needs the relevant DNS entries to be added before it can be merged." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [01:21:47] (03CR) 10Tim Starling: "Well, the apache config is OK in that the wikis will be served correctly from the new domains, but redirects will need to be added for b/c" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [01:24:35] Krinkle: https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&c=Bits+caches+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [01:25:26] https://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Bits+caches+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [01:25:33] Krinkle: https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&c=Bits+application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [01:25:47] https://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Bits+application+servers+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report [01:26:26] (03PS1) 10Tim Starling: Additional domains for Icdbfb2e74a [operations/dns] - 10https://gerrit.wikimedia.org/r/99017 [01:31:47] (03CR) 10Tim Starling: [C: 032] Additional domains for Icdbfb2e74a [operations/dns] - 10https://gerrit.wikimedia.org/r/99017 (owner: 10Tim Starling) [01:33:42] the bits app servers are going to disappear out of existence at this rate [01:34:15] hehe [01:34:41] ori-l: Somewhere in my Twitter stream in like early 2011 there's a Ganglia graph of the bits app servers CPU being cut in half [01:35:21] By changing ResourceLoaderUserModule::getModifiedTime() from return max( 0, max( map( 'mtime', $userJSPages ) ) ); to return max( 1, .... ); [01:35:24] (since 0 means "now") [01:36:26] heh [01:36:31] we've had a few overloads of the bits appservers, causing CSS delivery to be pretty much down [01:37:02] I'm not sure what to do about it [01:37:36] Mark was saying something about just throwing them into the general pool [01:37:38] well, load in general should be much reduced now, but it's definitely still fragile [01:38:00] it's quite possible that if the two pools were merged, the previous issues would have overloaded the whole appserver pool [01:38:21] then the site would have been even more down [01:38:44] maybe the bits varnishes just need to limit their backend connection count [01:39:41] I don't know, something clever needs to be done, anyway [01:39:43] should should also have monitoring for continuously increasing module mtimes [01:39:48] *we should [01:41:36] well, the issue on November 14 was not related to module mtime [01:43:07] well, for the ones I caused, I really don't get why 403s need a one-minute TTL [01:45:17] ori-l: Yeah we should monitor mtimes tracking within 1 minute of NOW() for like more than 5 minutes straight I suppose? That requires a fair bit of state-keeping probably though [01:45:52] there are lots of ways to overload the bits appservers, not just those two [01:46:11] you could have private responses, or you could have many URLs [01:48:40] (03CR) 10Tim Starling: "I added the DNS entries." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [01:49:33] if (req.url ~ "modules=startup") beresp.ttl += rand() % 10; [01:50:13] 10 * 60, rather [01:51:04] most bits app server request URLs are generated in JS by ResourceLoader based on the contents of the startup manifest [01:51:25] is it possible to have varnish return a 503 if the number of pending backend requests for a given URL exceeds some value? [01:51:58] 503s from bits wouldn't be very nice [01:52:03] It'd be better to return stale content if present [01:52:28] sure, if there is stale content [01:52:58] but if there is none, you have the choice of issuing 503s for the problematic URLs, or letting the whole backend pool overload and issuing 503s for everything, after a timeout [01:52:58] https://www.varnish-software.com/static/book/Saving_a_request.html [01:53:24] "When Varnish is in grace mode, it uses an object that has already expired as far as the TTL is concerned. There are several reasons this might happen, one of them being if a backend is marked as bad by a health probe." [01:54:31] that sounds like the right solution [01:54:48] well, it is the default, I think [01:54:55] but like I said, it only works if there is stale cached content [01:55:02] which is not guaranteed [01:55:21] for example, the responses may be private, or may be treated as private [01:56:00] 500s from apache will presumably not be cached [01:56:06] e.g. PHP timeouts [01:56:18] so the backend declaration specifies some URL that varnish should poll; if varnish's request fails, it marks the backend as sick and serves cached objects past their normal expiry [01:58:31] (03CR) 10Tim Starling: [C: 04-1] "$wmgWikimediaDatabaseLists is only populated on a conf cache miss, so normally, it will be empty." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/57173 (owner: 10Reedy) [01:58:38] hmm [01:59:49] is it possible to have varnish return a 503 if the number of pending backend requests for a given URL exceeds some value? [02:00:53] you could have the health check URL point to a URL that the bits app servers use to report pending requests [02:00:53] or even some separate web host that monitors the app servers [02:01:33] sounds complicated [02:10:58] just looking through the varnish source [02:13:44] there's apparently not a hashtable of pending requests keyed by URL or anything convenient like that [02:15:31] !log LocalisationUpdate completed (1.23wmf5) at Wed Dec 4 02:15:31 UTC 2013 [02:15:41] you would have to traverse a list of all sessions [02:15:47] Logged the message, Master [02:16:59] the backends have a list of connections, but I can't see a way to work out the URL for a given connection [02:20:29] (03PS10) 10Tim Starling: Add a sqldump script wrapper around mysqldump [operations/puppet] - 10https://gerrit.wikimedia.org/r/43844 (owner: 10Reedy) [02:20:52] (03CR) 10Tim Starling: [C: 032] Add a sqldump script wrapper around mysqldump [operations/puppet] - 10https://gerrit.wikimedia.org/r/43844 (owner: 10Reedy) [02:21:41] !log LocalisationUpdate completed (1.23wmf4) at Wed Dec 4 02:21:41 UTC 2013 [02:21:57] Logged the message, Master [02:23:19] TimStarling: this is what I had in mind: http://paste.tstarling.com/p/YsQEEX.html [02:23:36] it doesn't seem too complicated to me [02:27:04] probe.url is just the path part, you can't send a probe to some other host or port [02:27:37] it just uses the host/port of the backend [02:28:31] well, if the goal is to mark a backend as unhealthy before it is actually maxed and unable to respond [02:28:47] then we're targetting a point in time at which apache is still responsive [02:28:55] so it could just as well be a simple CGI script [02:29:18] and it doesn't have to be loadavg, it can be based on wc -l /proc/net/tcp or whatever [02:30:11] well, apache already has MaxClients and configurable listen backlog [02:30:58] but taking the whole of bits down is not really the feature I was looking for [02:31:14] letting it reach MaxClients will do that well enough [02:31:46] I would like it so that problematic (high request rate, slow) URLs are served 503s, while the rest of the site is still up [02:36:23] dunno, i have some hazy ideas but nothing definite. maybe strip out the RL timestamp from rl URLs in vcl_hash so that there is a always a grace response [02:37:00] well, not if the set of modules that are requested changes, but dunno [02:40:05] you could also have a canonical and ideally constant fallback URL for some basic JS / CSS that client-side ResourceLoader knows to default to if requests to bits are failing [02:40:29] wait wait wait, I know [02:40:36] use squid as varnish backend! [02:41:00] ;) [02:41:07] well, it's not completely absurd, i thought about adding another layer earlier [02:41:19] mark will kill you tho [02:41:21] you know the delay pool feature? [02:42:31] huh. yeah, that looks useful [02:44:57] gotta run, i'm afraid you'll have to do without my technical expertise [02:54:47] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 4 02:54:47 UTC 2013 [02:55:03] Logged the message, Master [03:48:57] (03PS1) 10Tim Starling: Re-add the docroot/secure directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99024 [03:50:37] Why is $wgCanonicalServer set to http:// for the private wikis? [03:50:45] That seems like a bug. [04:02:55] (03PS1) 10Tim Starling: secure.wikimedia.org ErrorDocument [operations/apache-config] - 10https://gerrit.wikimedia.org/r/99026 [04:04:01] what private wikis? [04:04:19] I see several with https: [04:05:34] TimStarling: I was looking at https://gerrit.wikimedia.org/r/#/c/53885/4/wmf-config/InitialiseSettings.php,unified [04:05:55] Hmm, perhaps out of date. [04:06:19] Seems so. [04:17:03] (03CR) 10Tim Starling: [C: 032] "Looks good. But needs manual rebase." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 (owner: 10Reedy) [04:17:04] (03CR) 10jenkins-bot: [V: 04-1] Move a lot of the miscellaneous wikis out of their own specific docroots [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 (owner: 10Reedy) [04:25:22] ori-l, let me know when you want to talk about JsonConfig - i want to figure out a few key points before implementing them [04:26:45] that's not agile, you know [04:26:58] ori-l, you? [04:27:00] or the config? [04:27:10] heh [04:27:13] anyways, what's up? [04:27:31] well, so i started porting zero config to a separate extension [04:28:03] the goal being Config:Proxies:Opera pages [04:28:18] or Proxy: [04:28:35] and JsonConfig ext would allow plugins that define this subspace [04:29:21] plugins would basically define a hook to handle validation [04:30:11] something like $wgConfigNs[NS_CONFIG]['Proxy'] = { validatorFunc } [04:30:35] well, see https://bugzilla.wikimedia.org/show_bug.cgi?id=44057 [04:30:44] some jerk committed to doing it and then ran off [04:31:20] ori-l, not a jerk, a busy individual :) [04:31:27] besides, not exactly what i need [04:31:34] i need validation + ability to change values [04:31:41] very specific [04:32:14] validator func would call my function with each field name, providing a callback that knows how to validate just that specific value [04:33:03] example: validator() { $config->check( 'ips', myIpValidatorFunc ) [04:33:58] well, I wouldn't write off ContentHandler just yet [04:34:13] the interface includes preSaveTransform [04:34:15] contenthandler is used everywhere around there :) [04:34:24] i'm using that too [04:34:34] it also has a notion of article parts that it calls 'section' [04:34:55] hmm, that might be useful [04:35:03] i'm kinda scared of touching it though :) [04:35:28] i should proly get wikidata ppl onboard [04:35:30] also from my experience with eventlogging, the set of constraints that JSON schema let you specify is rich enough to express the set of valid values that a property can take [04:35:56] i think it'll be overkill for most use-cases to have to write a validator function for every single field [04:36:25] JSON schema even has the concept of a regexp validator for strings, but I don't think Rob's library currently implements it [04:36:29] agree - if we can get a general case schema done, it would solve most usecases [04:37:14] one thing though - we should be able to specify custom manipulators - for example remove duplicates, sort, etc [04:38:00] right, but a pre-save transform for the configuration object as a whole is probably adequate, right? I mean, you could write a fancy transformer that inspects & changes specific properties [04:38:06] I wonder if that worked... [04:38:06] (03PS5) 10MZMcBride: Update wgServer, wgCanonicalServer for sub.subdomain wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [04:38:08] (03CR) 10jenkins-bot: [V: 04-1] Update wgServer, wgCanonicalServer for sub.subdomain wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [04:38:46] Closer, maybe. [04:38:48] ori-l, that's true, we could in theory separate the two [04:39:05] actually there are exceptions [04:39:18] the ability to edit JSON schema themselves online seems like a use-case that is specific to EventLogging; JSON schema doesn't even have to be JSON, really, since the PHP validator just decodes the object to an associative array [04:39:19] a frequent use case would be a new version [04:39:56] so you could treat the JSON schema spec as a notation for specifying fields & constraints via an associative array [04:40:11] what do you mean, new version? [04:40:57] example - your config has { "field" : "value" } and later you decide to support { "field": { "subfld1" : "value1", ... [04:41:09] so in validator you could rewrite one into another [04:41:28] dynamically, but that would happen every time config is parsed, not when its saved [04:41:37] when saving, you would save it in the new version [04:42:07] and this way all the logic that uses that field would not have to check which one it is [04:42:16] but treat it as a dict of dicts [04:42:59] hmm [04:43:04] (03PS3) 10Tim Starling: Switch www portals to using one docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91209 (owner: 10Reedy) [04:43:08] schema migrations are tricky [04:43:21] (03CR) 10Tim Starling: [C: 032] Switch www portals to using one docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91209 (owner: 10Reedy) [04:43:24] even though the older config on wiki will still stay unchanged. This might be highly useful for a huge number of config instances [04:44:09] true, they are tricky - but if you have all the transformation logic in the validation block (that gets executed every time config page is loaded), it will be ok [04:45:49] with EventLogging each event specifies the schema it validated against by revision ID, which is nice, because revisions are (mostly) immutable [04:46:24] maybe you could do something similar, like take the notion of an attached namespace (the way talk pages accompany article pages) [04:47:11] and somehow have the schema that describes the object be connected to it [04:47:33] well, i guess the relationship of objects to schemas is many-to-one, so that wouldn't work really [04:47:50] but the broader point that i'm trying to make is that you should have some way of versioning schema [04:47:57] or maybe not support migrations at all, or curb them somehow [04:48:02] otherwise it could easily get out of hand [04:49:27] it might make sense to just decline to provide any facility for schema migrations in the JsonConfig extension itself [04:50:22] i think schema migrations should be avoided if at all possible; if you can't practically avoid them then you can do it yourself by having your pre-save and pre-load methods do fancy things, but I don't think the JsonConfig extension itself should provide more structure than that [04:50:40] (03PS6) 10MZMcBride: Update wgServer, wgCanonicalServer for sub.subdomain wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [04:50:50] ori-l, not sure versioning will work in zero case - each user is identified by IP->ID mapping - we process it, and forget about it right thereafter. Nothing is stored long term except configs themselves - and the page name is the ID [04:50:52] * Elsie waits for jenkins' scorn. [04:51:21] i guess we should facilitate it (since its already done by zero anyway) [04:51:34] but in the basic settings use a simple json schema validation [04:51:52] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 303 seconds [04:51:55] so basically you map the user's IP address to a config object that tells you various things about how you should handle the request? [04:52:04] yep [04:52:20] and the handling is done by the same extension that stored it in the first place [04:52:22] PROBLEM - MySQL Slave Delay on db1046 is CRITICAL: CRIT replication delay 313 seconds [04:52:34] ori-l, maybe we should get off this channel :) [04:52:39] servers don't like us [04:55:31] !log running sync-apache to deploy I730ec0bf8c388cada17b46497d87146f1e4ded1a [04:55:47] Logged the message, Master [04:57:43] (03PS7) 10MZMcBride: Update wgServer, wgCanonicalServer for sub.subdomain wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [04:57:47] (03PS1) 10Yurik: Corrected comment for FlaggedRevs deployment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99028 [04:59:10] (03CR) 10Tim Starling: "Deployed." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91209 (owner: 10Reedy) [05:00:09] Is nomcom dead? [05:01:56] !log add index on db1048 otrs.ticket_history.article_id to stop daily slave lag [05:02:10] Logged the message, Master [05:02:43] (03PS8) 10MZMcBride: Update wgServer, wgCanonicalServer for sub.subdomain wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/53885 (owner: 10Reedy) [05:03:19] Okay, I think that's finished. [05:03:29] I'll submit a separate changeset for nomcom. [05:04:38] noncom was killed on October 24. [05:05:15] nomcom [05:05:32] DNS was killed, it looks like the wiki itself has been dead for much longer. [05:07:58] (03PS1) 10MZMcBride: nomcom is dead. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99030 [05:45:15] (03PS3) 10MZMcBride: Create "Draft" namespace on the English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 [05:54:13] (03CR) 10TTO: Create "Draft" namespace on the English Wikipedia (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [05:56:01] (03CR) 10Legoktm: Create "Draft" namespace on the English Wikipedia (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [05:59:13] (03PS4) 10MZMcBride: Create "Draft" namespace on the English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 [05:59:59] (03CR) 10MZMcBride: Create "Draft" namespace on the English Wikipedia (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [06:28:14] PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:30:04] RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active [06:34:57] (03CR) 10Ori.livneh: [C: 031] [WIP] Add configuration for Wikimania Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [06:45:04] PROBLEM - Disk space on tungsten is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=96%): [06:48:04] RECOVERY - Disk space on tungsten is OK: DISK OK [06:55:10] (03PS1) 10Ori.livneh: Graphite: use package's default storage dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/99032 [06:56:29] (03CR) 10Ori.livneh: [C: 032] Graphite: use package's default storage dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/99032 (owner: 10Ori.livneh) [08:27:47] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [09:12:01] Elsie: oh trust me you don't want to even go near vixie's cron [09:12:38] Elsie: I fixed a vulnerability 6-7 years ago and my eyes were bleeding [09:12:39] apergos: on beta, Parsoid now self updates from the master branch :-D [09:12:54] apergos: had a crazy bug which caused it to show the articles from production, got it fixed yesterday \O/ [09:13:45] woah that is crazy [09:13:49] but yay for having it working [09:14:27] some path issues, the daemon was launched using a code base that missed a configuration file, so it was failing back to prod :/ [09:14:45] ouch! [09:19:56] paravoid: were is gerrit-wm? [09:20:13] no idea [09:20:37] * matanya looks at the floor of the channel [09:26:58] (03PS1) 10Hashar: beta: update Parsoid dependencies only on changes [operations/puppet] - 10https://gerrit.wikimedia.org/r/99052 [09:26:59] (03PS1) 10Hashar: beta: missing docstring in autoupdater [operations/puppet] - 10https://gerrit.wikimedia.org/r/99053 [09:27:17] matanya: gerrit-wm got replaced with grrrit-wm which runs on labs [09:27:26] gerrit-wm used to be some python hooks in Gerrit itself [09:28:09] hashar: so why grrrit-wm doesn't report my new push? [09:28:20] was/is broken maybe? [09:30:28] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:58] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:14] hi ops [09:31:41] we're seeing errors coming from the search API that Wikipedia via Text in Kenya is using. [09:31:49] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:58] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:58] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:08] Is this a known issue? [09:32:18] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 85484 bytes in 1.366 second response time [09:32:21] not incredibly regular but enough to show up on my graphs [09:32:40] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64982 bytes in 0.794 second response time [09:32:48] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 65006 bytes in 0.799 second response time [09:32:48] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64870 bytes in 0.803 second response time [09:32:49] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 85498 bytes in 0.828 second response time [09:33:09] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [09:33:28]