[02:11:31] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 21.43% of data exceeded the critical threshold [500.0] [02:12:32] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 9 below the confidence bounds [02:14:04] !log LocalisationUpdate completed (1.24wmf5) at 2014-05-26 02:13:01+00:00 [02:14:15] Logged the message, Master [02:18:37] (03PS3) 10Tim Landscheidt: Tools: Install cmake [operations/puppet] - 10https://gerrit.wikimedia.org/r/135318 (owner: 10Petrb) [02:20:54] (03CR) 10Tim Landscheidt: [C: 031] Tools: Install cmake [operations/puppet] - 10https://gerrit.wikimedia.org/r/135318 (owner: 10Petrb) [02:25:42] !log LocalisationUpdate completed (1.24wmf6) at 2014-05-26 02:24:39+00:00 [02:25:46] Logged the message, Master [02:34:31] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [03:11:08] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon May 26 03:10:02 UTC 2014 (duration 10m 1s) [03:11:13] Logged the message, Master [03:51:41] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [07:01:29] jouncebot: nice one ;) [07:28:09] hello [07:31:17] <_joe_> hi hashar [07:32:38] I am in Paris this week :D [07:32:58] with family, so will be randomly connected :( [07:33:17] <_joe_> oh well, I'm with my desk in a newly-emptied hous [07:33:21] _joe_: have you got any issue with the Jenkins integration of puppet catalog compiler ? [07:33:28] shah empty houses are the best [07:33:40] that gives you the feeling of having a well organized place [07:33:41] <_joe_> hashar: no it works like a charm [07:33:48] <_joe_> not really [07:33:55] <_joe_> tons of garbage around :) [07:34:08] <_joe_> we moved to a new house yesterday, but no DSL there [07:34:53] obviously [07:48:49] (03CR) 10Hashar: [C: 031] Tools: Install cmake [operations/puppet] - 10https://gerrit.wikimedia.org/r/135318 (owner: 10Petrb) [07:49:13] (03PS2) 10Hashar: inserted iotop to toollabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/135185 (owner: 10Petrb) [07:50:35] (03CR) 10Hashar: [C: 031] inserted iotop to toollabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/135185 (owner: 10Petrb) [08:30:51] Can't connect to the database. [08:30:51] Error: Can't connect to MySQL server on 'db1001.eqiad.wmnet' (111) [08:30:51] Is your database installed and up and running? [08:30:51] Do you have the correct username and password selected in localconfig? [08:31:01] (trying to access bugzilla) [08:31:07] working now [08:38:49] (03CR) 10Hashar: "The wikis have not be added to the Jenkins job that takes care of updating the databases causing: Bug 65677 - Hindi Beta Wikipedia broken:" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/129210 (owner: 10Chad) [09:16:07] <_joe_> !log repaired table bugzilla.bugs_fulltext on db1001 as it was marked as crashed [09:16:12] Logged the message, Master [09:17:08] !log bugzilla.bugs_fulltext bug was {{bug|65762}} [09:17:12] Logged the message, Master [09:17:12] \O/ [09:51:26] (03Restored) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89002 (owner: 10Hashar) [09:51:30] (03PS8) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89002 [09:52:23] (03PS9) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89002 [11:03:52] (03PS1) 10Filippo Giunchedi: grant cscott deployment access [operations/puppet] - 10https://gerrit.wikimedia.org/r/135418 [12:42:30] (03PS1) 10Faidon Liambotis: Switch DMARC reports to dmarc-report@ [operations/dns] - 10https://gerrit.wikimedia.org/r/135423 [12:43:02] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Switch DMARC reports to dmarc-report@ [operations/dns] - 10https://gerrit.wikimedia.org/r/135423 (owner: 10Faidon Liambotis) [12:58:36] (03CR) 10Alexandros Kosiaris: [C: 04-1] puppet_compiler: add ferm rule to allow web access (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/135050 (owner: 10Giuseppe Lavagetto) [13:03:03] (03CR) 10Matanya: puppet_compiler: add ferm rule to allow web access (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/135050 (owner: 10Giuseppe Lavagetto) [13:50:31] (03CR) 10Hashar: "Note the new Jenkins job https://integration.wikimedia.org/ci/job/operations-puppet-tox-data_admin_lint/5/console which seems to confirm " [operations/puppet] - 10https://gerrit.wikimedia.org/r/135418 (owner: 10Filippo Giunchedi) [14:01:32] (03PS1) 10John F. Lewis: Move-categorypages permission changes on fawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/135426 (https://bugzilla.wikimedia.org/65728) [14:11:37] (03CR) 10Calak: [C: 031] Move-categorypages permission changes on fawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/135426 (https://bugzilla.wikimedia.org/65728) (owner: 10John F. Lewis) [14:41:20] hashar: good job on the admin yaml jenkins job! [14:41:36] ahhh [14:41:40] I am pleased you like it :] [14:41:53] I thought that admin stuff will be prone to errors [14:42:06] and Chase did all the work (i.e. write the actual linter) [14:42:13] maybe i should post about it on ops list [14:49:55] hashar: can you give some clue about InitialiseSettings.php "wiki groups"? [14:51:22] Specifically, at line 7598 I don't understand if that will be enough to replace/complement the defaults for those wikis https://gerrit.wikimedia.org/r/#/c/134400/5/wmf-config/InitialiseSettings.php [14:52:05] Nemo_bis: hmm I would prefer not to have to mess with those settings :-] [14:52:17] we lack tests :-( [14:52:54] I asked just in case you remembered the tricks of groupOverrides2 :) [14:53:26] that one is old :] [14:53:29] I asked about ways to test in https://wikitech.wikimedia.org/wiki/Talk:Configuration_files [14:53:38] As your experience with configs :P [14:53:47] at one point I wanted to migrate all that PHP cruft to some light format such as CSV / YAML or whatever [14:53:55] and generate the PHP array [14:56:47] (03CR) 10Nemo bis: "If doubts over usage of groupOverrides2 are a blocker, I can always use groupOverrides instead. It just requires adding some more + here a" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134400 (owner: 10Nemo bis) [15:08:21] (03PS1) 10Yuvipanda: dynamicproxy: Make proxy depend on cert being installed [operations/puppet] - 10https://gerrit.wikimedia.org/r/135434 [15:08:35] Coren: ^ minor fix to prevent double puppet runs required when adding a cert [15:09:37] (03CR) 10coren: [C: 032] "This makes exactly 1.0713 senses." [operations/puppet] - 10https://gerrit.wikimedia.org/r/135434 (owner: 10Yuvipanda) [15:10:26] Coren: ty [15:17:18] (03Abandoned) 10Krinkle: Embed $wgCopyrightIcon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125927 (owner: 10Ori.livneh) [15:33:11] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Mon 26 May 2014 12:32:59 PM UTC [15:33:21] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Mon May 26 15:33:13 UTC 2014 [15:55:55] (03PS1) 10Edenhill: Added %D and %T formatters (request time) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/135441 [16:15:00] akosiaris: ping? Any particular reason why postgres access is request-only now? [16:43:51] (03PS5) 10Ottomata: Adding $deployable_networks variable in network.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125184 [16:44:41] (03CR) 10Ottomata: "Bryan, let's merge this week sometime. Wanna get it out of my gerrit review queue. :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/125184 (owner: 10Ottomata) [16:46:57] (03CR) 10Ottomata: "Hm, was this not supported before?" [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/135441 (owner: 10Edenhill) [16:53:45] YuviPanda: pong. Meaning unlike mysql on labs? Mostly cause it was installed for the toolserver migration needs and not much else. So no automatic ways of creating users exist like for mysql@labs [16:56:39] akosiaris: hmm, right. but if that work were to be done, are there any other objections to opening up postgres users? [16:58:51] not in theory no. Performance wise we may have an issue though. map tile generation is quite IO expensive and I think it would be best to not create unnecessary performance problems to that. [16:59:42] this of course works in the opposite direction to. map tile generation could possibly choke other applications [16:59:49] too* [17:02:42] akosiaris: hmm, right. [17:03:07] (03CR) 10Edenhill: "Nope, this wasnt in the varnishncsa I based the format support on." [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/135441 (owner: 10Edenhill) [17:14:27] (03CR) 10Ottomata: [C: 032 V: 032] "Ah ok, cool." [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/135441 (owner: 10Edenhill) [17:14:44] (03CR) 10Ottomata: "Popular, eh?! Who else is using varnishkafka?" [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/135441 (owner: 10Edenhill) [17:43:33] (03PS1) 10Yuvipanda: toollabs: Initial work for the mongo role [operations/puppet] - 10https://gerrit.wikimedia.org/r/135442 [17:46:32] (03PS2) 10Yuvipanda: toollabs: Initial work for the mongo role [operations/puppet] - 10https://gerrit.wikimedia.org/r/135442 [17:52:47] ori: you created the mongo module in ops/puppet, right? Am setting up a mongo instance for tools, and am starting from there (and modifying to suit) [17:55:07] YuviPanda: yep [18:10:34] ori: hmm, I am considering just rm -rf ing the mongo module and just using a template file for conf. the older config format is deprecated, newer one is yaml based, and I don't know how / if we can specify nested settings easily in puppet hashes and have them convert to yaml. [18:11:00] ori: ok, can't rm -rf it but can at least not use it [18:11:00] yes we can [18:11:19] ori: oh? infinitely nested? [18:11:36] oh, right. erb is just ruby [18:11:43] it's trickier than that [18:11:44] so I should be able to just yaml.dumps or equivalent [18:11:56] the order of keys in a hash is undefined [18:12:03] and in practice randomized [18:12:20] so if you just rely on the built-in yaml to_s it'll generate log churn [18:12:28] because the order of keys will change from one puppet run to the next [18:12:30] didn't we sort them by keys before? [18:12:34] to prevent this? [18:12:37] yes, we have ordered_json [18:12:47] since json is a subset of yaml, ordered_json works just fine [18:13:00] oh, right. valid json is valid yaml, I forgot that too [18:13:16] git grep for 'ordered_json' in the repo and you'll see examples [18:15:27] ori: ordered_json looks cool! [18:18:01] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:34] indeed tungsten isn't very happy, looking [18:20:32] tungsten isn't very strong today :p [18:20:52] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.019 second response time [18:23:45] ori: I suppose I'll have to migrate the other uses of mongo class as well. doing that now. [18:23:45] indeedly, it came back behaving by itself [18:27:13] ori: oh, ugh. can't actually just switch everything, since precise mongo is 2.0.4 and doesn't support yaml, and for tools I'd be running latest (official debs, labsdebrepo) [18:27:29] right, i figured you'd have some parameter [18:27:39] if set, render this template, if not, render that template [18:27:39] ori: that switches versions? [18:27:46] ori: hmm, that should work too [18:27:56] ori: I wonder what that should be called tho [18:28:07] is the filename the same? [18:28:18] yeah [18:28:21] let me verifyu [18:28:43] ori: yeah [18:28:48] /etc/mongo.conff [18:29:25] without the f [18:29:53] well, we don't provision mongo anywhere, and if someone wanted to start using it in production, we'd probably take the time to actually package a more recent version (or perhaps the version in trusty is already using the yaml format) [18:30:04] in short, i think it's okay to remove support for the old config format [18:30:33] (03PS1) 10coren: Labs: puppetize replica-addusers [operations/puppet] - 10https://gerrit.wikimedia.org/r/135445 [18:30:56] ori: hmm, so should I just remove the old code that is used in statistics and eventlogging? [18:31:41] (03CR) 10jenkins-bot: [V: 04-1] Labs: puppetize replica-addusers [operations/puppet] - 10https://gerrit.wikimedia.org/r/135445 (owner: 10coren) [18:34:09] (03PS2) 10coren: Labs: puppetize replica-addusers [operations/puppet] - 10https://gerrit.wikimedia.org/r/135445 [18:36:26] (03CR) 10coren: [C: 031] "This basically puppetizes status quo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135445 (owner: 10coren) [18:37:32] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 21.43% of data exceeded the critical threshold [500.0] [18:38:01] ori: ok, so the mongo class instantiation on eventlogging is also kinda useless since the consumer has been disabled for a long time no9w [18:39:47] (03PS1) 10Yuvipanda: eventlogging: Remove unused MongoDB code [operations/puppet] - 10https://gerrit.wikimedia.org/r/135446 [18:39:51] ori: ^ [18:52:45] (03CR) 10Ori.livneh: [C: 031] eventlogging: Remove unused MongoDB code [operations/puppet] - 10https://gerrit.wikimedia.org/r/135446 (owner: 10Yuvipanda) [18:54:41] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 8 below the confidence bounds [18:55:57] (03PS1) 10Ori.livneh: Add rsyslog module and port existing usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/135447 [18:57:11] (03CR) 10jenkins-bot: [V: 04-1] Add rsyslog module and port existing usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/135447 (owner: 10Ori.livneh) [18:57:31] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data exceeded the critical threshold [500.0] [18:59:42] (03PS2) 10Ori.livneh: Add rsyslog module and port existing usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/135447 [19:27:42] Coren: +2 https://gerrit.wikimedia.org/r/#/c/135446/ maybe? cleaning up the mongodb module. has a +1 from ori [19:36:37] o/~ Hey, I just met you, and this is crazy, but here's my changeset, so +2 maybe. o/~ [19:36:59] (03CR) 10coren: [C: 032] "Seems sane." [operations/puppet] - 10https://gerrit.wikimedia.org/r/135446 (owner: 10Yuvipanda) [19:37:14] Coren: haha [19:37:26] Coren: putting that into quips [20:02:31] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [20:26:17] (03CR) 10JanZerebecki: Improve nginx TLS/SSL settings. (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [20:35:27] (03PS3) 10JanZerebecki: Improve nginx TLS/SSL settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) [20:38:41] (03PS4) 10JanZerebecki: Improve nginx TLS/SSL settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) [20:50:16] (03PS1) 10Yuvipanda: toollabs: Install pastebinit [operations/puppet] - 10https://gerrit.wikimedia.org/r/135493 [20:50:21] Coren: trivial patch? ^ [20:51:04] (03CR) 10coren: "Wouldn't that need a configuration to be generally useful?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135493 (owner: 10Yuvipanda) [20:51:51] Coren: it works by default, just puts things on ubuntu paste. I don't think putting things that might be useful for debugging tools *on* a server hosted by tools is a good idea. also, liangent is now testing a config that can potentially paste to tools. paste, so we can make an alias for it later. [20:52:49] The only case where I see a problem with pasting to labs is when you're trying to paste something broken about the pasting itself. :-) [20:53:44] I'd rather not have tools that send data outside of labs by default if it's avoidable; especially from the command line where it's not unheard-of to have people accidentally paste PII [20:53:51] YuviPanda: don't forget to configure http://tools.wmflabs.org/paste/ as the pastebin service [20:54:04] Yeah, what matanya said. [20:54:11] Coren: ah, the PII argument does make sense [20:54:29] Coren: have a working config now, let me puppetize [20:54:53] I mean, I have nothing against posting stuff on other pastebins; but I'd rather the default stay on labs. Presumably, that's configurable per-user with a dotfile right? [20:55:08] Coren: yeah [20:55:18] Coren: let me amend the patchset [20:55:33] you can just use a skel in aet [20:55:37] etc [20:57:45] (03PS4) 10Yuvipanda: Tools: Alias tools.wmflabs.org to internal webproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/123149 (https://bugzilla.wikimedia.org/54052) (owner: 10Tim Landscheidt) [21:04:41] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [21:08:26] (03PS1) 10Ottomata: [WIP] Add CDH5 support, drop CDH4 support [operations/puppet/cdh4] (cdh5) - 10https://gerrit.wikimedia.org/r/135494 [21:08:29] (03CR) 10JanZerebecki: "PS4 which compared to PS1 removes HSTS and disables DHE can now be tested on https-test.wmflabs.org" [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [21:08:47] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add CDH5 support, drop CDH4 support [operations/puppet/cdh4] (cdh5) - 10https://gerrit.wikimedia.org/r/135494 (owner: 10Ottomata) [21:16:19] (03CR) 10JanZerebecki: [C: 031] Remove wiktionary.wikipedia.org from rewrites as it is not in DNS. [operations/apache-config] - 10https://gerrit.wikimedia.org/r/92799 (owner: 10Reedy) [21:34:27] YuviPanda: Could you add "Bug: 50935" to the pastebinit commit message, please? [21:36:40] scfc_de: sure [22:06:12] (03PS3) 10Ori.livneh: Add rsyslog module and port existing usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/135447 [22:14:56] (03CR) 10Parent5446: "Considering this is the exact recommended ciphersuite from Mozilla, I have no doubts about its security impact. The only question is, like" [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [22:21:44] (03PS4) 10Ori.livneh: Add rsyslog module and port existing usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/135447 [23:04:27] (03PS2) 10Tim Landscheidt: Tools: Install pastebinit [operations/puppet] - 10https://gerrit.wikimedia.org/r/135493 (https://bugzilla.wikimedia.org/50935) (owner: 10Yuvipanda) [23:04:52] scfc_de: I'm adding a config file to it, so didn't push yet. will make sure to add bug number in next ps [23:24:06] (03PS3) 10Yuvipanda: toollabs: Install pastebinit [operations/puppet] - 10https://gerrit.wikimedia.org/r/135493 (https://bugzilla.wikimedia.org/50935) [23:24:11] Coren: scfc_de ^ [23:25:21] (03CR) 10coren: [C: 032] "I approve of this message." [operations/puppet] - 10https://gerrit.wikimedia.org/r/135493 (https://bugzilla.wikimedia.org/50935) (owner: 10Yuvipanda) [23:26:22] YuviPanda: Just wanted to write it down somewhere so I don't have to remember it :-). [23:26:28] scfc_de: :) [23:39:15] ori: oh, didn't realize we had puppet stdlib in ops/puppet now [23:40:21] ori: I don't know if merge() handles recursive hashes, though [23:41:19] (03PS3) 10Yuvipanda: toollabs: Initial work for the mongo role [operations/puppet] - 10https://gerrit.wikimedia.org/r/135442 [23:41:21] (03PS1) 10Yuvipanda: mongo: Support newer yaml style configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/135499 [23:42:44] (03CR) 10jenkins-bot: [V: 04-1] toollabs: Initial work for the mongo role [operations/puppet] - 10https://gerrit.wikimedia.org/r/135442 (owner: 10Yuvipanda) [23:42:55] (03CR) 10jenkins-bot: [V: 04-1] mongo: Support newer yaml style configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/135499 (owner: 10Yuvipanda) [23:48:33] aude: Touched the scripts (no log as logmsgbot was gone) [23:48:46] for wmf5 only [23:49:16] (03PS4) 10Yuvipanda: toollabs: Initial work for the mongo role [operations/puppet] - 10https://gerrit.wikimedia.org/r/135442 [23:49:18] (03PS2) 10Yuvipanda: mongo: Support newer yaml style configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/135499 [23:50:32] that's fine [23:56:01] (03PS1) 10Tim Landscheidt: Tools: Fix pastebinit configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/135500 [23:56:25] (03CR) 10Tim Landscheidt: [C: 04-1] "Needs to be tested first." [operations/puppet] - 10https://gerrit.wikimedia.org/r/135500 (owner: 10Tim Landscheidt)