[00:09:27] (03PS1) 10Yuvipanda: dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 [00:15:46] hey, can someone look up for me the version of jetty on one of solr* hosts, please? [00:23:22] paravoid: heads up; we're launching another onslaught [00:25:06] mwalker: did you find the cause for the 4xx? [00:25:52] no; I think it's in a banner -- but I cant prove that -- I'm guessing the 4xx will return; but I sort of want to make sure [00:26:09] It could have been in that particular banner; or it could be in our backend workflow [00:26:10] (03PS2) 10Yuvipanda: dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 [00:27:03] okay, I'll get a sample and try to find outliers [00:27:03] (03CR) 10jenkins-bot: [V: 04-1] dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 (owner: 10Yuvipanda) [00:27:41] Is it just me, or does https://test.wikipedia.org/wiki/Main_Page not have any CSS? [00:28:27] not just you [00:29:18] paravoid: where is the tap for the 4xx plot? [00:29:28] is it all cluster wikis? or everything served through varnish? [00:29:54] the latter [00:30:09] (03PS3) 10Yuvipanda: dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 [00:31:03] (03CR) 10jenkins-bot: [V: 04-1] dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 (owner: 10Yuvipanda) [00:33:20] (03PS4) 10Yuvipanda: dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 [00:34:19] paravoid: I'm not seeing it happen again; so it might have just been that banner... do you know what the unit on the Y axis is? count / sec? count / min? [00:35:12] 4xx/min [00:35:44] what you mean by "that banner"? [00:36:57] the centralnotice banner / object we were serving at the time [00:36:59] it's variable [00:37:10] (03PS5) 10Yuvipanda: dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 [00:37:11] Coren: can you merge? ^ [00:37:15] Coren: tested! [00:38:42] anyone else to give me a +2 on a labs specific patch that's not being used anywhere yet? [00:41:39] (03PS1) 10Springle: depool slaves for packages upgrade, including mariadb 5.5.34 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97656 [00:42:06] (03CR) 10Springle: [C: 032] depool slaves for packages upgrade, including mariadb 5.5.34 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97656 (owner: 10Springle) [00:43:07] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [00:43:24] !log springle synchronized wmf-config/db-eqiad.php 'depool slaves for packages upgrade' [00:43:40] Logged the message, Master [00:45:36] (03PS6) 10Yuvipanda: dynamicproxy: Make 404 fallback servers configurable [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 [00:45:38] * yuvipanda_ gently pokebegs ori-l [00:46:33] I don't know anything about the setup [00:47:26] it isn't really running anywhere, and I tested it, but I'll just wait up for one of the labs folks to show up tomorrow I guess [00:47:32] thanks anyway ori-l [00:47:53] let me read the patch [00:47:56] * yuvipanda_ puts a pox on the bigass one-size-fits-all puppet-repo [00:48:13] ori-l: no trailing commas in this one :) [00:48:18] ori-l: and thanks :) [00:49:09] it looks safe to merge, but I think you're better off waiting. I don't have the attention span to meaningfully review it, and it looks substantial enough that it could very plausibly be improved by having someone with some context look at it [00:49:22] * paravoid concurs fwiw [00:49:40] alright :) [00:49:55] working... slightly odd hours with the puppet repo is just frustrating. [00:50:32] the most recent patchset is 5 minutes old [00:50:44] ori-l: I edited the commit message [00:51:05] ok, 40 minutes old :P [00:51:27] ori-l: okay, maybe I'm being whinier than I should be :P [00:56:24] (03PS5) 10Dzahn: bugzilla module - WIP [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 [01:00:38] ori-l: btw, in your nginx patch I think you mentioned that you tested the patch on vagrant. [01:00:51] ori-l: did you just clone the module into vagrant and test it out? [01:01:04] s/clone/cp/ [01:01:07] PROBLEM - Puppet freshness on db1034 is CRITICAL: No successful Puppet run for 3d 15h 42m 42s [01:01:09] yes [01:01:32] ori-l: ah, okay. [01:01:41] nice that that was modular enough :) [01:02:18] ori-l: should perhaps split it out to its own module, when you have the time (might also make the license situation less murky) :) [01:02:58] (03CR) 10coren: [C: 032] "Yuvi says it works. If it breaks, blame him. :-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97655 (owner: 10Yuvipanda) [01:03:16] thanks Coren :D [01:03:28] PROBLEM - mysqld processes on db1051 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [01:04:07] oops, should have silenced [01:07:57] (03PS6) 10Dzahn: bugzilla module - WIP [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 [01:09:03] (03CR) 10Dzahn: "fixed issues found in labs testing and some comments by Alex" (039 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [01:11:27] (03CR) 10Dzahn: "see inline comments on PS4, many comments have been addresses but not all. though now i can say as long as you comment the require "passwo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [01:15:58] PROBLEM - Puppet freshness on db1034 is CRITICAL: No successful Puppet run for 0d 0h 1m 0s [01:16:38] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Tue Nov 26 01:16:32 UTC 2013 [01:16:58] PROBLEM - Puppet freshness on db1034 is CRITICAL: No successful Puppet run for 0d 0h 0m 25s [01:17:58] PROBLEM - Puppet freshness on db1034 is CRITICAL: No successful Puppet run for 0d 0h 1m 0s [01:19:38] RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Tue Nov 26 01:19:27 UTC 2013 [01:20:56] (03PS1) 10Springle: warm up slaves after package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97661 [01:21:24] (03CR) 10Springle: [C: 032] warm up slaves after package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97661 (owner: 10Springle) [01:21:59] hey paravoid, how would i disable puppet freshness checks for professor? [01:22:12] (03CR) 10Dzahn: bugzilla module - WIP (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [01:22:17] it's what you and akos were suggesting initially, i think it was right in hindsight [01:22:30] !log springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade' [01:22:45] Logged the message, Master [01:22:45] oh, it's 3:20 AM for you. don't answer:P [01:23:14] ori-l: disable in what way? really not known by icinga or just not reporting anywhere besides web ui [01:23:42] not reporting anywhere besides web ui [01:24:02] ori-l: got icinga-admin login? [01:24:20] no. i can look in the usual places [01:24:32] https://icinga-admin.wikimedia.org/icinga try it [01:24:52] oh, that worked [01:25:01] :) [01:25:17] sweet, and it's already disabled [01:25:20] so, you can either do 'scheduled downtime' or just "disable notifications" [01:25:22] or both [01:25:30] so probably pvoid or akos had the good sense to just go ahead and do it [01:25:37] and you can do so for the host and /or all the services on it [01:25:51] it should give you a checkbox for that if you from the host overview page [01:26:05] and you then put an RT ticket in the comment field and send :P) [01:26:08] do what? [01:26:31] * AaronSchulz wonders why loginwiki is on s3 instead of s7 [01:26:59] now it's just 2 SPOFs (loginwiki/centralauth) ;) [01:27:21] experience has shown that if we lose a shard, we lose everything [01:27:41] so it might matter less in practice [01:27:46] not that I disagree [01:28:15] and it'd be nice to not lose appservers altogether when a shard gets overloaded [01:41:51] marktraceur: when you were doing your upload wizard stuff earlier; what exactly were you doing? [01:42:12] ... or more precisely; was it something that would've caused a bunch of HTTP 4xx's? [01:45:16] (03CR) 10GWicke: "Alternative approach (gzip support in Parsoid) is implemented in https://gerrit.wikimedia.org/r/#/c/97652/. It looks like that works, but " [operations/puppet] - 10https://gerrit.wikimedia.org/r/97647 (owner: 10GWicke) [01:45:31] mwalker: It, um...probably shouldn't have? [01:45:42] mwalker: Do you have logs of what the 4xxs are trying to reach? [01:46:01] Wikimedia Errors? [01:46:04] Getting HTTP 503 [01:46:07] paravoid: ^ [01:46:09] Wuuuuh oh. [01:46:37] Krenair: on what site? enwiki seems to be fine [01:46:52] enwiki... it's fine now. [01:48:04] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [01:50:24] eh, WTF is https://nah.wikipedia.org/wiki/Tehtlahtol - it seems to be causing a lion's share of PHP timeouts [01:51:58] (03PS1) 10Springle: slaves to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97663 [01:55:43] (03PS1) 10Ori.livneh: Configure gdash to be served by nginx::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/97664 [01:56:05] (03CR) 10Springle: [C: 032] slaves to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97663 (owner: 10Springle) [01:56:19] (03PS1) 10Tim Starling: wikipedia.com.il -> wikipedia.co.il [operations/apache-config] - 10https://gerrit.wikimedia.org/r/97665 [01:56:58] !log springle synchronized wmf-config/db-eqiad.php 'slaves to full steam' [01:57:11] (03CR) 10Ori.livneh: [C: 032] Configure gdash to be served by nginx::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/97664 (owner: 10Ori.livneh) [01:57:22] Logged the message, Master [01:57:46] springle: did you see the conversation between me and ori yesterday about the pc1001 overload? [01:57:58] TimStarling: no [01:58:03] channel? [01:58:05] here [01:58:27] I was intending to summarize it and reply on ops@, but I haven't gotten to it yet [01:58:32] I can grab the log for you, hang on [01:59:02] the TL;DR is that it didn't seem like MediaWiki was running a higher-than-normal rate of queries on that database host [01:59:29] there was no increase in replace, delete or select query rate that we could see, in the ganglia data [01:59:53] in fact there was a reduction in query rate as the server slowed down [02:00:32] the bit that confused me was the spike in mysql_bytes_received [02:00:40] yeah, that was a metric bug [02:00:56] same as the spikes in all the other mysql_* metrics [02:01:08] it only affects a single sample, when the server restarts [02:01:08] oh [02:01:45] someone should probably fix that [02:01:53] if a counter resets to 0, gmetad assumes it was due to overflow [02:02:05] gmetad is not at fault [02:02:32] oh, i see -- it's the gmond module [02:02:33] ganglia gives you a few different ways to deal with such issues [02:02:43] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [02:02:54] it's the fault of the metric script for not implementing one of them [02:03:47] OK, I'll file a bug for it and assign it to myself [02:05:14] I love that there's a pc1001 and a cp1001 [02:05:24] (03CR) 10Tim Starling: [C: 032 V: 032] Generate redirects.conf [operations/apache-config] - 10https://gerrit.wikimedia.org/r/96438 (owner: 10Tim Starling) [02:05:38] given that, i'm tempted to reduce pc100[123] max_connections. plus aaron's SqlBagOStuff commit last week to reduce txn times would presumably help [02:06:05] ori-l: what does cp stand for anyway? [02:06:14] it was fairly obvious when we called them sq* [02:06:48] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (201387) [02:06:52] you mean, the ambiguity was confined to 'squid' and 'sql'? :) [02:07:05] (03CR) 10Tim Starling: [C: 032 V: 032] wikipedia.com.il -> wikipedia.co.il [operations/apache-config] - 10https://gerrit.wikimedia.org/r/97665 (owner: 10Tim Starling) [02:07:48] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [02:07:49] I mean, given the function of the server, it was easy to guess where the name came from [02:08:26] (03PS1) 10Ori.livneh: Fix template file path in Gdash role [operations/puppet] - 10https://gerrit.wikimedia.org/r/97667 [02:10:34] where are the parsercache dbs configured at the mediawiki level? not db-eqiad.php [02:11:30] $pcServers in CommonSettings.php [02:13:04] thanks [02:13:18] load is balanced via consistent hashing of keys [02:13:52] but the slow query log on pc1001 shows a wide distribution of keys [02:14:25] atm pc100[123] only replicate to the old pmtpa boxes. is there a need for an eqiad slave each? if only for fail over [02:14:54] definitely [02:15:38] $pcTemplate says 'type' => 'mysql' . is there another parsercache type? [02:16:08] i mean, besides gwicke's cassandra ideas [02:16:11] !log LocalisationUpdate completed (1.23wmf4) at Tue Nov 26 02:16:11 UTC 2013 [02:16:26] Logged the message, Master [02:17:20] AFAIK any of the classes implementing the object cache interface can be used; current this includes redis, memcached, APC, a few other things [02:17:22] that type is the $type parameter to DatabaseBase::factory(), i.e. the DBMS [02:17:39] right [02:17:40] so it can be pgsql, sqlite, oracle, etc. [02:18:31] but you see that there is also a class option which is set to SqlBagOStuff [02:18:58] that can instead be RedisBagOStuff etc. to get non-SQL storage [02:19:55] the last item from the investigation yesterday is that cpu_wio climbed to ~5% starting on the 21st, which could correspond to 1 of 24 cores being maxed; I didn't see anything unusual around that time in syslog. It doesn't correlate with a cron job or a Puppet config change [02:20:46] it could have been a slow query, or many slow queries [02:20:58] they would only be logged in the slow query log if they finished before the server was restarted [02:22:17] That seems like a bug. [02:22:51] heh, good luck with that [02:23:23] The query has to finish successfully to be logged as a slow query? [02:23:25] oh, maybe it's not clear, we are talking about the slow query log which is part of MySQL [02:23:29] Isn't there a query slayer? [02:23:40] yes, springle [02:24:13] !log LocalisationUpdate completed (1.23wmf5) at Tue Nov 26 02:24:12 UTC 2013 [02:24:18] presumably you would be laughed at if you asked Oracle to ensure queries running at the time of a server kill were entered into the slow query log [02:24:28] Logged the message, Master [02:24:30] The server kill I understand. [02:24:36] But it seems reasonable to log long-running queries. [02:24:48] Especially if they don't get logged if they're killed by something other than a restart. [02:24:51] I don't know MySQL internals well enough if they'd be in the binlog [02:24:52] Which was really my question, I think. [02:25:08] !log deploying new generated redirects.conf [02:25:12] there was nothing in pc1001 processlist runing longer than the 1000+ SqlBagOStuff txns siting waiting to COMMIT. the other 4000 or so were running slow but as a result of whatever allowed all those writers to pile up [02:25:21] Logged the message, Master [02:25:39] innodb was fighting to get undo slots and unable to get ahead again [02:26:20] hence wanting to decrease max_connections. normal traffic is only hundreds concurrent, so it should be safe enough [02:26:49] maybe domas has an idea, he's online in #mediawiki_security [02:28:58] PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 3d 0h 10m 56s [02:30:32] He's going to recommend using mysql-facebook. ;-) [02:32:32] pc1001 still is the facebook branch [02:32:36] although the 5.1 one [02:33:02] Ah, nice. [02:33:09] Facebook: pwn3d [02:45:34] PROBLEM - Disk space on cerium is CRITICAL: DISK CRITICAL - free space: /mnt/data 15977 MB (3% inode=99%): [03:10:04] (03PS3) 10Tim Starling: Normalise the path part of URLs in the text frontend [operations/puppet] - 10https://gerrit.wikimedia.org/r/96941 [03:12:49] !cp is caching proxy (squid or varnish) [03:12:49] Key was added [03:13:39] (03CR) 10Tim Starling: [C: 032] Normalise the path part of URLs in the text frontend [operations/puppet] - 10https://gerrit.wikimedia.org/r/96941 (owner: 10Tim Starling) [03:14:08] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Nov 26 03:14:08 UTC 2013 [03:14:24] Logged the message, Master [03:16:42] !pc is parser cache (mysql) [03:16:42] Key was added [03:18:31] RECOVERY - Disk space on cerium is OK: DISK OK [03:19:51] <^d> !es is external store (and sometimes elasticsearch, but we try not to) [03:19:51] Key was added [03:19:54] <^d> mutante: ^ [03:20:32] :) [03:21:03] !mw [03:21:06] !mw is what we never call mw but appserver [03:21:06] Key was added [03:21:32] <^d> But MW is MediaWiki ;-) [03:21:59] !mw is mediawiki but we usually call them appserver [03:22:00] This key already exist - remove it, if you want to change it [03:22:09] !mw del [03:22:09] Successfully removed mw [03:22:12] !mw is mediawiki but we usually call them appserver [03:22:12] Key was added [03:23:51] !osm is open street map [03:23:51] Key was added [03:24:01] !tmh is timed media handler [03:24:01] Key was added [03:24:39] !srv are the old appservers, now "mw" [03:25:21] !srv is old app servers, now "mw" [03:25:22] Key was added [03:29:30] !log deploying varnish normalize_path (I687f6773) [03:29:44] Logged the message, Master [03:38:59] (03PS1) 10Springle: Reduce max_connections for parsercache dbs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/97671 [03:40:18] (03CR) 10Springle: [C: 032] Reduce max_connections for parsercache dbs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/97671 (owner: 10Springle) [03:47:30] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [03:48:27] (03PS1) 10Springle: Add mysqld process icinga check. [operations/puppet] - 10https://gerrit.wikimedia.org/r/97672 [03:49:57] (03CR) 10Springle: [C: 032] Add mysqld process icinga check. [operations/puppet] - 10https://gerrit.wikimedia.org/r/97672 (owner: 10Springle) [04:04:19] (03PS1) 10Springle: depool slaves for package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97674 [04:04:54] (03CR) 10Springle: [C: 032] depool slaves for package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97674 (owner: 10Springle) [04:05:38] !log springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade' [04:05:52] Logged the message, Master [04:13:09] (03PS1) 10MZMcBride: Create "Draft" namespace on the English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 [04:15:13] (03CR) 10Dzahn: role and module structure for ishmael (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 (owner: 10Dzahn) [04:29:38] (03CR) 10Dzahn: "since this broke once before i'd really appreciate if somebody else could take a look here and confirm" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91209 (owner: 10Reedy) [04:32:53] (03CR) 10Dzahn: "if anyone has good or new ideas here.. please go ahead, i'm currently not sure what to do with it" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/65443 (owner: 10Dzahn) [04:34:32] (03CR) 10Dzahn: "this would move dsh to a module, should be ok, but we don't seem to use that monitoring check (yet) somebody started and how much longer a" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96413 (owner: 10Dzahn) [04:35:21] (03CR) 10Dzahn: "not needed anymore but a reminder for me to change the update method to use curl" [operations/puppet] - 10https://gerrit.wikimedia.org/r/95317 (owner: 10Dzahn) [04:36:37] (03CR) 10Dzahn: "the related RT #5928 is stalled. JeremyB? odder? wanna pick it up?" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/88705 (owner: 10Dzahn) [04:38:08] (03CR) 10Dzahn: "thanks for the info, so.. hmm.. what do we want regarding those NS changes then?" [operations/dns] - 10https://gerrit.wikimedia.org/r/86659 (owner: 10Dzahn) [04:41:22] mutante: you have mail... [04:42:28] (03CR) 10Dzahn: "hrmm.. meanwhile it's just" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96935 (owner: 10Dzahn) [04:44:58] (03CR) 10Dzahn: "so did i poossibly some too fast then that work again meanwhile? wanna just amend this change?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96935 (owner: 10Dzahn) [04:45:01] jeremyb: got it, thanks [04:45:03] and good night [04:47:11] you too :) [04:49:36] PROBLEM - Puppet freshness on tungsten is CRITICAL: No successful Puppet run for 3d 19h 31m 10s [04:50:34] (03PS1) 10Springle: depool the correct slave this time! [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97677 [04:51:01] (03CR) 10Springle: [C: 032] depool the correct slave this time! [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97677 (owner: 10Springle) [04:51:45] !log springle synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade' [04:52:00] Logged the message, Master [05:03:55] (03CR) 10Jeremyb: "waiting on legal, repoked them" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/88705 (owner: 10Dzahn) [05:11:09] (03CR) 10Swalling: [C: 04-1] "Like I said on the bug, I think there are unanswered questions regarding functionality and permissions for this namespace that are (for no" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [05:20:32] (03CR) 10Ori.livneh: [C: 032] Fix template file path in Gdash role [operations/puppet] - 10https://gerrit.wikimedia.org/r/97667 (owner: 10Ori.livneh) [05:22:08] RECOVERY - Puppet freshness on tungsten is OK: puppet ran at Tue Nov 26 05:22:00 UTC 2013 [05:29:51] (03PS1) 10Ori.livneh: tungsten: serve gdash on gdash.wm.o vhost [operations/puppet] - 10https://gerrit.wikimedia.org/r/97678 [05:29:58] PROBLEM - Puppet freshness on rhodium is CRITICAL: No successful Puppet run for 3d 3h 11m 57s [05:31:31] (03PS1) 10Springle: warm up slaves after package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97679 [05:31:53] (03CR) 10Springle: [C: 032] warm up slaves after package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97679 (owner: 10Springle) [05:32:38] !log springle synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade' [05:32:54] Logged the message, Master [05:34:22] (03CR) 10Ori.livneh: "err: Failed to apply catalog: Could not find dependency File[/etc/init/ocg-collection.conf] for Service[ocg-collection] at /etc/puppet/man" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 (owner: 10Mwalker) [05:35:11] thanks ori-l -- the plan is to fix that tomorrow [05:36:07] (03PS1) 10Ori.livneh: role::ocg: fix typo in config file name [operations/puppet] - 10https://gerrit.wikimedia.org/r/97680 [05:36:10] carpe diem [05:37:09] (03CR) 10Ori.livneh: [C: 032] tungsten: serve gdash on gdash.wm.o vhost [operations/puppet] - 10https://gerrit.wikimedia.org/r/97678 (owner: 10Ori.livneh) [05:37:19] (03CR) 10Ori.livneh: [C: 032] role::ocg: fix typo in config file name [operations/puppet] - 10https://gerrit.wikimedia.org/r/97680 (owner: 10Ori.livneh) [05:38:05] that works too [05:38:06] :) [05:38:53] you wouldn't also happen to have the power to create git repositories too would you? [05:39:03] mutante: 5343 could use an update too (seems to now be available for anyone to register) but I don't want to say anything while the squatter's still listed on CC [05:39:05] RECOVERY - Puppet freshness on rhodium is OK: puppet ran at Tue Nov 26 05:38:57 UTC 2013 [05:40:13] mwalker: what's the repo? [05:40:54] operations/ocg-config [05:42:02] oh [05:42:08] i'd prefer to leave that to jeff [05:42:22] if i create it with the default ACLs, you won't be able to push to it anyway [05:42:39] jeff doesn't know how to do it / may not have privs [05:42:40] if i give you push rights, I'd be overstepping what I'm supposed to do [05:42:50] oh [05:43:02] if you at least create it; I can get jeff to give me privs [05:43:26] or; we can leave it as an operations only +2 repo [05:43:32] something to hash out tomorrow [05:43:52] https://dpaste.de/33mU/raw [05:44:01] specifically err: /Stage[main]/Role::Ocg::Collection/Service[ocg-collection]/ensure: change from stopped to running failed: Could not find init script for 'ocg-collection' [05:45:01] luckily I do have shell on rhodium! [05:45:04] * mwalker rummages [05:45:51] I wonder if it needs a /etc/init.d/ocg-collection script as well as an upstart config [05:46:14] no, it doesn't [05:46:27] Upstart-job does that, but it's not necessary [05:46:29] hurm; initctl list shows the service [05:46:37] it needs provider => upstart, [05:46:44] ahhh [05:46:46] on the service resource [05:47:00] what should the short description of the repo be? [05:47:18] it's usually a single sentence [05:47:25] Configuration files for Offline Content Generation servers? [05:47:37] Haughty Capital Letters [05:47:43] :) [05:47:51] it is the name of the role [05:48:10] could mention something about PDF rendering and mathoid [05:48:32] so, Configuration files for offline content generation servers (pdf rendering, mathoid) [05:49:33] https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/ocg-config [05:49:35] oh [05:49:41] I already went with the Haughty Capital Letters [05:50:34] Jeff_Green: fwiw, I ran: ssh -p 29418 gerrit gerrit create-project --require-change-id --owner=opssoftware --parent=operations/software --description='"Configuration files for Offline Content Generation servers"' operations/software/ocg-config [05:51:31] is there an existing gerrit group for ocg stuff? [05:52:26] we were just using extension-collection so far [05:52:40] ori-l: i don't know [05:52:49] morning [05:52:50] Jeff_Green: what are you still doing up! [05:53:04] Jeff_Green: sorry, I didn't mean to ping; I figured you'd get it tomorrow sometime [05:53:06] trying to detangle the fundraising mysql privs snafu [05:53:18] morning pv [05:53:25] oh :'( [05:58:00] (03PS1) 10Mwalker: Stating that OCG is an upstart job and setting config repo [operations/puppet] - 10https://gerrit.wikimedia.org/r/97681 [05:58:29] (03PS1) 10Springle: slaves to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97682 [05:58:55] (03CR) 10Springle: [C: 032] slaves to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97682 (owner: 10Springle) [05:59:45] !log applying workaround for Ganglia XSS https://github.com/ganglia/ganglia-web/issues/218 [05:59:49] !log springle synchronized wmf-config/db-eqiad.php 'slaves to full steam after package upgrade' [05:59:54] no CVE yet [05:59:58] Logged the message, Master [06:00:14] Logged the message, Master [06:01:15] (03PS1) 10Ori.livneh: Add missing trailing semicolon to Gdash Nginx config [operations/puppet] - 10https://gerrit.wikimedia.org/r/97683 [06:02:30] (03PS2) 10Mwalker: Stating that OCG is an upstart job and setting config repo [operations/puppet] - 10https://gerrit.wikimedia.org/r/97681 [06:02:38] silly whitespace [06:03:17] (03CR) 10Ori.livneh: [C: 04-1] "Small nit" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97681 (owner: 10Mwalker) [06:03:31] oh [06:04:08] (03CR) 10Ori.livneh: [C: 032] Add missing trailing semicolon to Gdash Nginx config [operations/puppet] - 10https://gerrit.wikimedia.org/r/97683 (owner: 10Ori.livneh) [06:04:21] ori-l: what was your small nit? [06:05:31] the whitespace, and ensure not being the first property [06:06:35] ah; ok; I'll flip the ensure [06:06:37] but, um, [06:06:52] * mwalker waits [06:07:03] you asked for operations/ocg-config, rather than operations/software/ocg-config [06:07:16] so the mistake was mine. you shouldn't change the manifest [06:07:40] oh; I figured you had a reason to put it under software [06:07:48] I just guessed at where it should live [06:08:04] paravoid: ... where should a configuration repo live? [06:08:06] gah, I rushed it without thinking. [06:08:13] I should delete the repo. [06:09:44] (03PS1) 10Ori.livneh: Add missing trailing semicolon to Gdash Nginx config [operations/puppet] - 10https://gerrit.wikimedia.org/r/97684 [06:11:18] I deleted it [06:11:30] good to get faidon's two cents on repo namespace [06:13:17] (03CR) 10Ori.livneh: [C: 032] Add missing trailing semicolon to Gdash Nginx config [operations/puppet] - 10https://gerrit.wikimedia.org/r/97684 (owner: 10Ori.livneh) [06:13:29] fatal: Project not found: operations/software/ocg-config [06:13:29] fatal: The remote end hung up unexpectedly [06:13:31] :-P [06:19:49] hurm; that doesn't sound good [06:20:01] but jeremyb; maybe you know -- where should a configuration repo live? [06:20:16] Isn't there a wmf-config directory? [06:20:47] there is; but it doesn't seem like that's where this should live [06:20:51] but... maybe it should [06:21:17] a single file (or maybe two files); for a server that wont host a MediaWiki install [06:21:27] mwalker: jeremyb was just confirming i deleted it [06:21:43] shouldn't it be in puppet? [06:22:06] לַיְלָה טוֹב [06:22:14] you too [06:22:18] I was about to say that. [06:22:23] !log pwercycled labstore1001, unreachable, nothing on mgmt console [06:22:38] Logged the message, Master [06:22:53] mwalker: https://git.wikimedia.org/repositories/ [06:23:01] operations/puppet probably, though I'm not really paying attention in here. [06:23:14] neither am I, evidently [06:23:15] :/ [06:23:16] If that ocg repo was deleted, it's still lingering on GitHub and git.wm.o. [06:23:24] RECOVERY - Host labstore1001 is UP: PING OK - Packet loss = 16%, RTA = 0.29 ms [06:23:33] i deleted it on github [06:23:38] git.wm.o is just a caching thing [06:23:55] Fair enough. [06:24:07] Strangely, my GitHub news feed only announces the creation. [06:24:25] I toyed with putting it into puppet as an erb; but that didn't seem correct to have service configuration in with server configuration [06:25:14] apache conf, puppet, mediawiki conf are all directly under operations [06:25:35] how does parsoid get configured now? or does it not have configuration? [06:26:28] good night [06:26:30] :) [06:26:32] good night [06:26:38] mwalker, there's a distinction drawn between modules and roles [06:27:15] a module -- in its platonic form, at least -- provides a set of generic abstractions for provisioning and configuring a particular application [06:28:53] a role declares how a module is configured on some particular node or nodes [06:29:57] so; should ocg actually be written as a module? [06:30:04] yes [06:30:10] ideally you'd have a generic ocg module [06:30:26] ah; ok -- I'll work that in [06:30:33] and you'd configure it by instantiating the module from a role file in manifests/role/ocg.pp or whatever [06:31:05] configuration files are typically either managed by passing all the specific values as parameters to the module [06:31:15] and having the module itself generate the config file based on the parameters [06:31:27] or, when it is impractical to do it that way [06:31:28] this makes it much easier for a labs instance owner to re-use your module by setting up their own role with their own config params [06:31:39] (for example) [06:32:12] yeah [06:33:10] when it's impractical to have the module generate the configuration files (because the set of configuration options supported by the application is too large for each one to be represented as a parameter) [06:33:23] you can have the role deploy files via the files/ dir and templates via the templates/ dir [06:33:57] it's entirely appropriate for files in either of those locations to contain site-specific configs [06:35:02] TL;DR: blah blah puppet blah blah blah [06:35:30] puppet == evil [06:35:30] heh, I wanted to bring up those ocg issues myself, but I was thinking of doing it after the sprint is over as to not block it [06:35:33] thanks ori-l :) [06:35:48] I'd rather do it right the first time [06:35:58] possibly the second time [06:36:14] in case of large ice cube in the way of the first time [06:36:16] :P [06:37:16]