[00:00:21] (03CR) 10Rush: [C: 032 V: 032] trusty friendly phabricator [operations/puppet] - 10https://gerrit.wikimedia.org/r/143510 (owner: 10Rush) [00:03:00] (03Abandoned) 10Yuvipanda: toollabs: Don't collect NFS stats [operations/puppet] - 10https://gerrit.wikimedia.org/r/143236 (owner: 10Yuvipanda) [00:04:26] (03Abandoned) 10Rush: phabricator: use apache::site, not apache::vhost [operations/puppet] - 10https://gerrit.wikimedia.org/r/143378 (owner: 10Ori.livneh) [00:04:48] chasemp: :) [00:10:04] (03PS1) 10Rush: my.cnf.erb syntax fixup [operations/puppet] - 10https://gerrit.wikimedia.org/r/143523 [00:12:22] (03CR) 10Rush: "example:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143523 (owner: 10Rush) [00:13:27] (03CR) 10Dzahn: [C: 031] my.cnf.erb syntax fixup [operations/puppet] - 10https://gerrit.wikimedia.org/r/143523 (owner: 10Rush) [00:14:12] (03CR) 10Rush: [C: 032] my.cnf.erb syntax fixup [operations/puppet] - 10https://gerrit.wikimedia.org/r/143523 (owner: 10Rush) [00:15:07] (03CR) 10BryanDavis: [C: 031] "Applied on deployment-bastion: apache vhost looks good and trebuchet works." [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [00:15:54] (03PS1) 10Dzahn: replace deprecated erb template variable syntax [operations/puppet] - 10https://gerrit.wikimedia.org/r/143526 [00:17:24] chasemp, mutante: bd808's patch is the last apache::vhost, could one of you review? it's scoped to beta where it is already applied [00:19:14] (03CR) 10Rush: labs: role::deployment - port apache::vhost to apache::site (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [00:19:33] ah, that's a good point [00:19:37] bd808: i'll amend [00:22:10] (03PS4) 10Ori.livneh: labs: role::deployment - port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [00:22:27] PROBLEM - HTTP on francium is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 700 bytes in 0.003 second response time [00:22:27] PROBLEM - Memcached on francium is CRITICAL: Connection refused [00:22:27] (03PS5) 10Ori.livneh: labs: role::deployment - port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [00:23:50] (03CR) 10Rush: [C: 031] "cool" [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [00:23:54] (03PS6) 10Ori.livneh: labs: role::deployment - port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [00:24:18] chasemp: i can't merge it myself; i can only +2 my own patches [00:24:33] ok merging sorry [00:24:44] (03CR) 10Rush: [C: 032 V: 032] labs: role::deployment - port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [00:24:58] wee, thanks [00:25:30] Ah. Good. I didn't like the $::fqdn thing etiher [00:25:57] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Puppet has 1 failures [00:26:23] (03PS1) 10Dzahn: deprecated syntax in icinga checkcommands.cfg.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/143527 [00:27:45] (03CR) 10Rush: [C: 031] deprecated syntax in icinga checkcommands.cfg.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/143527 (owner: 10Dzahn) [00:29:21] (03PS1) 10Ori.livneh: apache: get rid of apache::vhost [operations/puppet] - 10https://gerrit.wikimedia.org/r/143528 [00:31:51] (03PS1) 10Dzahn: deprecated syntax in mysql/generic_my.cnf.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/143529 [00:33:02] (03PS2) 10Dzahn: deprecated syntax in mysql/generic_my.cnf.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/143529 [00:35:50] (03PS1) 10Rush: legalpad to radon [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 [00:37:29] (03PS2) 10Rush: legalpad to radon [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 [00:38:03] chasemp: what do you mean by "#designated to iridium can be below it soon " [00:38:26] yeah fixed the lingo there [00:38:35] I realized my brain didn't catchup to my hands [00:38:59] but essentially used hostname matching for radon to do the same below it for iridium, even tho iridium isn't in there yet [00:39:03] was what I meant to say [00:39:41] (03CR) 10Dzahn: legalpad to radon (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 (owner: 10Rush) [00:40:08] chasemp: ah,hm, so iridium will be nonlegal-ph ? [00:40:26] heh, I was going to to just make iridum teh default namespace [00:40:30] phabricator- [00:40:42] the db's for radon are named like [00:40:47] phlegal-maniphest [00:40:47] etc [00:40:54] phlegal is cleared with springle btw [00:41:00] imho it should be 2 role classes to avoid "if ($::fqdn" inside a role [00:41:10] like, we already apply role to just the right nodes... [00:41:11] k, so yeah I wasn't sure [00:41:26] would it be [00:41:27] (03PS6) 10Yuvipanda: labs: Enable diamond PuppetAgent collector on all nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 [00:41:31] role::phabricator::radon? [00:41:38] role::phabricator::legalpad [00:41:40] role::phabricator::legal [00:41:43] the lattetr [00:41:46] and then the default? [00:41:48] dont use a node name in there [00:41:55] role::phabricator::phabricator seemed odd [00:41:58] so I didn't go there [00:42:11] role::phabricator::default? [00:42:15] role::phabricator::production ? [00:42:20] yea, or that [00:42:27] I'm sold will amend [00:43:35] it could even be.. role::phabricator::legal inherits role::phabricator::base [00:43:48] but not trying to make it more complicated.. need food still:) [00:43:51] well I don't love that because legal doesn't really inheret from base [00:43:57] heh understood [00:44:21] yea, just make 2 roles in that one role file [00:46:35] (03PS1) 10Ori.livneh: apache: improve docs; provision apache::def before apache::{mod_conf,conf} [operations/puppet] - 10https://gerrit.wikimedia.org/r/143532 [00:47:26] paravoid: that about wraps it up for my ::apache adventures [00:47:29] i feel pretty good about the result [00:47:38] i hope opsen find it nice to use [00:47:55] I haven't seen it yet [00:48:00] who will migrate us off webserver.pp? :) [00:48:17] (03CR) 10Dzahn: [C: 031] apache: improve docs; provision apache::def before apache::{mod_conf,conf} [operations/puppet] - 10https://gerrit.wikimedia.org/r/143532 (owner: 10Ori.livneh) [00:49:00] dunno, whomever is up for it. i have to turn to other things. i did get rid of apache_site, apache_mod, apache::vhost, and a bunch of other monstrosities [00:49:22] (03PS3) 10Rush: legalpad to radon [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 [00:49:24] it should be pretty straightforward [00:49:30] mutante: thanks [00:50:46] wanna do https://gerrit.wikimedia.org/r/#/c/143528/ as well? it's easy to verify that apache::vhost is gone, you can grep for it [00:52:05] (03CR) 10Dzahn: legalpad to radon (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 (owner: 10Rush) [00:55:45] (03CR) 10Dzahn: [C: 031] "looks like it's not used in prod, yea, grepped for it" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143528 (owner: 10Ori.livneh) [00:56:19] (03CR) 10Ori.livneh: [C: 032] apache: get rid of apache::vhost [operations/puppet] - 10https://gerrit.wikimedia.org/r/143528 (owner: 10Ori.livneh) [00:56:29] (03CR) 10Ori.livneh: [C: 032] apache: improve docs; provision apache::def before apache::{mod_conf,conf} [operations/puppet] - 10https://gerrit.wikimedia.org/r/143532 (owner: 10Ori.livneh) [00:57:25] mutante: :)))) [00:57:35] i may have a little ceremony now and release a dove and some balloons into the sky [00:58:06] * YuviPanda gives ori a Tuna sandwich [00:58:54] ori: add some flash drives with Korean WP :) [00:59:05] heh [00:59:24] it applied correctly on zirconium, random app server, and tin [01:00:33] ori: congrats, that entire refactoring [01:00:41] checked antimony, no issues [01:00:55] (03PS7) 10Yuvipanda: labs: Enable MinimalPuppetAgent collector on all nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 [01:01:00] chasemp: ^ I wonder if I did a puppet goofup [01:01:03] should chasemp put those variables inside the role class? [01:01:15] in the diamond::collector::minimalpuppetagent [01:01:34] mutante: fwiw I tested that does work [01:01:48] and I want those variables in more than one role class so it seems weird do dupe them for each [01:01:52] idk [01:02:03] * YuviPanda goes to test his patch [01:02:10] chasemp: ok, then just .. you want to use the new rolename in site.pp now [01:02:26] oops thought I fixed it honestly, coming up [01:03:38] (03PS4) 10Rush: legalpad to radon [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 [01:04:46] (03CR) 10Dzahn: [C: 031] "let's add base::firewall , but that's for tomorrow :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 (owner: 10Rush) [01:05:25] YuviPanda: not sure what you mean by puppet goofup [01:06:01] (03PS5) 10Rush: legalpad to radon [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 [01:06:09] (03CR) 10Rush: [C: 032 V: 032] legalpad to radon [operations/puppet] - 10https://gerrit.wikimedia.org/r/143531 (owner: 10Rush) [01:09:00] (03CR) 10Rush: [C: 04-1] "few thoughts...instead of" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 (owner: 10Yuvipanda) [01:09:18] chasemp: weirdly, I'm running into permission issues reading that yaml file as an unprevilaged user [01:10:47] hmmm should be world readable [01:11:02] chasemp: yeah, but even an ls /var/lib/puppet fails for me, and looking at the perms it shouldn't [01:11:03] ah but not on precise? [01:11:13] I am on precise. [01:11:19] precise: -rw-r----- 1 root root (that I see) [01:11:24] oh [01:11:26] trusty: [01:11:26] -rw-r--r-- 1 root root [01:11:33] idk why the diff, puppet version? [01:12:01] chasemp: maybe I should slip in something to make sure it's readable? [01:12:21] I would rather put in the header that it requires puppet agent version X [01:12:26] and then restrict it to trusty hosts in puppet [01:12:35] uh, well, tools can't use it... [01:12:42] do we have puppet3 backported to precise? [01:12:50] that is a question I don't know, I thought yes? [01:12:56] _joe_ is your guy there [01:13:14] chasemp: there are no trusty hosts in tools, and I don't think we're going to migrate for a few months at least. [01:13:36] we must have puppet3 on precise somehow, see f.e. host iron [01:13:46] Version: 3.4.3-1~ubuntu12.04.1 [01:13:49] interesting [01:13:55] puppet is 3.4.3 but it's precise [01:13:58] but now really out :) [01:13:59] chasemp: I think it's an ensure => latest problem [01:14:20] it's Version: 2.7.11-1ubuntu2.7 on the precise box I'm checking [01:14:31] chasemp: yeah, but apt-cache show tells me a 3.4 exists [01:14:31] in prod [01:14:42] chasemp: It's not installed by default, but it's available. [01:14:47] chasemp: see "iron" [01:14:58] (It is installed by default in Labs though) [01:15:52] YuviPanda: I think your next step is to see about mass ensure latest? [01:16:06] but I don't know why it isn't done now, etc etc [01:16:18] and I wonder what'll break :D [01:16:28] scumbag puppet writes a status files......not user readable [01:16:45] chasemp: can't I just set permissions on that file and let it be for this patch, and do the mass upgrade later? there are 250 hosts affected by this change... [01:17:07] I wouldn't personally be in favor of changing perms for a file managed by a deb outside of the deb [01:17:24] hmm [01:17:45] I understand your thought, I just....that's a slippery slope [01:17:46] chasemp: hmm,a ctually [01:17:53] chasemp: the box I'm on actually has puppet 3.4 [01:18:03] chasemp: and... still permission errors?! [01:18:20] yeah...wtf [01:18:38] -rw-r--r-- 1 root root 599 Jul 2 01:13 last_run_summary.yaml [01:18:46] drwxr-xr-t 3 puppet puppet 4096 Jul 2 01:13 state [01:18:49] what's the 't' bit? [01:19:20] sticky bit [01:19:26] https://en.wikipedia.org/wiki/Sticky_bit [01:19:28] http://www.linuxdevcenter.com/pub/a/linux/lpt/22_06.html [01:19:36] beat me and better link [01:19:38] touche [01:19:43] YuviPanda: Sticky. On directories, it means only the owner of a file can unlink it [01:19:49] ah, right [01:19:53] shouldn't actually matter here [01:20:02] It's silly, 'cause s is something else. [01:20:08] YuviPanda: I think I've seen where a new install vs. an upgrade can be permissions silly [01:20:11] sometimes [01:20:17] try installing from scratch the package [01:20:22] and then try upgrading it, etc [01:20:23] yuvipanda@graphite-test:~$ ls /var/lib/puppet [01:20:27] ls: cannot open directory /var/lib/puppet: Permission denied [01:20:35] chasemp: even ls reports it, so something weirder is happening [01:20:53] oh [01:20:56] I can access it fine now [01:20:59] wtf? [01:21:01] * YuviPanda is so confused atm [01:21:12] is it locked when it's running an update? [01:21:28] alice in wonderland moment here maybe [01:22:06] [2014-07-02 01:21:00,022] [Thread-68] graphite.graphite-test.puppetagent.puppetagent.time_since_last_run 444 1404264060 [01:23:35] chasemp: ghost in the machine? [01:24:04] chasemp: also do you think this metric would be useful in prod? [01:24:09] I don't think so because the precise box I'm looking at is still bad perms [01:24:29] chasemp: right, but all labs boxen seem to have puppet 3.4.3 [01:24:40] is good point [01:24:41] Coren: ^ is that assessment right? I checked a few precise boxes and they all have 3.4.3 [01:25:04] Yeah, all the labs boxen should be 3.4.3 [01:25:19] chasemp: let me move things around [01:25:31] * YuviPanda found about 4 boxes with stale pids due to /var/log exhaustion a few days ago [01:26:04] Yeay diamond? [01:26:37] Coren: heh :) [01:27:01] Coren: it shouldn't do that anymore, since labs rotates logs more aggressively now. but really, we shouldn't have 2G /vars [01:28:00] I beg to differ; we shouldn't have things that logs GB of stuff in days rather. [01:28:21] Unless it's actually expected/needed for the use in which case you then allocate /var/log accordingly. :-) [01:28:59] Coren: it was logging about 400M over 5 days, and since most instances are at about 1.3G full normally they went over. but yeah, this particular instance was diamond's fault, but I still think we should put biglogs everywhere [01:29:04] By definition, GB of logs for something you aren't actually trying to debug means you're logging too much stuff at to low a level and any signal is drowned. [01:29:21] indeed. [01:29:36] Coren: not defending diamond :) [01:29:39] well, in this case at least [01:30:09] it was a silly runaway log, should be cool now [01:30:40] chasemp: updating to reflect your comments [01:31:20] chasemp: shouldn't it be files/collector/ rather than collectors, to match manifests? [01:31:51] (03CR) 10TTO: "I love that this change has five +1s! It's such a happy sight." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/141100 (https://bugzilla.wikimedia.org/58932) (owner: 10TTO) [01:32:21] YuviPanda: the soul of semantics :) [01:32:28] :D [01:32:33] idk I think ppl are using plural that I've seen [01:32:49] but I'm not going to -1 becuase of it [01:32:57] (03PS8) 10Yuvipanda: labs: Enable MinimalPuppetAgent collector on all nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 [01:33:06] chasemp: I want to be consistent with manifests, so we change both or leave 'em be [01:33:16] * YuviPanda tests [01:35:31] YuviPanda: probably should throw in the header of the collector that it requires puppet version 3 or greater [01:35:37] prevent much confusion later [01:35:44] chasemp: yeah, makes sense [01:37:34] chasemp: hmm, I see it in diamond.log but not in graphite [01:37:47] what is it? [01:38:33] [2014-07-02 01:14:00,006] [Thread-40] graphite.graphite-test.puppetagent.puppetagent.time_since_last_run 24 1404263640 [01:38:35] the actual metrics [01:38:41] also I wonder if my code is goofing up somehow. [01:39:07] aaah, I'm not [01:39:13] unix timestamps are in seconds [01:39:25] OH DEAR GOD I'VE BEEN WRITING JAVA FOR TOO LONG TO EXPECT MILLISECONDS [01:40:13] epoch time :) [01:40:24] (03PS9) 10Yuvipanda: labs: Enable MinimalPuppetAgent collector on all nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 [01:40:39] you can watch the creation log on the graphite box to see if they are being created [01:40:50] yeah that's what I'm doing now [01:41:40] chasemp: uh oh, weird. I don't actually see it in the whisper files :| [01:43:42] so you see it logged, and other metrics show up in graphite [01:44:01] I would give it a minute? graphite creation tends to be throttled, unsure of the lab config [01:44:11] (03PS1) 10Rush: mediawiki oauth for phab [operations/puppet] - 10https://gerrit.wikimedia.org/r/143549 [01:44:18] chasemp: hmm, alright. [01:44:32] (03CR) 10jenkins-bot: [V: 04-1] mediawiki oauth for phab [operations/puppet] - 10https://gerrit.wikimedia.org/r/143549 (owner: 10Rush) [01:44:45] (03PS2) 10Rush: mediawiki oauth for phab [operations/puppet] - 10https://gerrit.wikimedia.org/r/143549 [01:46:32] (03CR) 10Rush: [C: 032 V: 032] "this was otherwise approved and merged, just moving the radon tag up to implement SUL" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143549 (owner: 10Rush) [01:46:53] chasemp: aha, found my bug. [01:47:49] (03PS10) 10Yuvipanda: labs: Enable MinimalPuppetAgent collector on all nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 [01:48:29] YuviPanda: have to step away I'll look if you still need eyes in the morning? [01:48:31] getting late here [01:48:50] chasemp: heh, 7 AM here :) yeah, I'll get someone to merge and if not poke you tomorrow [01:48:54] chasemp: thanks for the help/advice! [01:49:14] I should go to sleep soon [01:50:37] Coren: can you merge ^? works now :) [01:51:44] gah, no. [01:51:46] this is weird [01:52:39] ah, right. so the previous, buggy patch's metrics just showed up, and the new one's will in a while [01:52:52] (03PS11) 10Yuvipanda: labs: Enable MinimalPuppetAgent collector on all nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 [01:56:33] (03CR) 10Yuvipanda: [C: 031] "Tested. Metric are coming in now \o/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143193 (owner: 10Yuvipanda) [01:57:50] * YuviPanda goes to sleep [02:15:56] !log LocalisationUpdate completed (1.24wmf10) at 2014-07-02 02:14:53+00:00 [02:16:04] Logged the message, Master [02:27:28] !log LocalisationUpdate completed (1.24wmf11) at 2014-07-02 02:26:24+00:00 [02:27:33] Logged the message, Master [02:57:39] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 2 02:56:33 UTC 2014 (duration 56m 32s) [02:57:43] Logged the message, Master [03:00:43] !log rebooting lead [03:00:48] Logged the message, Master [03:02:37] PROBLEM - Host lead is DOWN: PING CRITICAL - Packet loss = 100% [03:02:57] RECOVERY - Host lead is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [03:04:08] (03PS1) 10Faidon Liambotis: Add role::mail::mx to lead [operations/puppet] - 10https://gerrit.wikimedia.org/r/143554 [03:04:59] (03CR) 10Faidon Liambotis: [C: 032] Add role::mail::mx to lead [operations/puppet] - 10https://gerrit.wikimedia.org/r/143554 (owner: 10Faidon Liambotis) [03:05:34] (03CR) 10Faidon Liambotis: [V: 032] Add role::mail::mx to lead [operations/puppet] - 10https://gerrit.wikimedia.org/r/143554 (owner: 10Faidon Liambotis) [03:08:57] PROBLEM - DPKG on lead is CRITICAL: DPKG CRITICAL dpkg reports broken packages [03:09:37] PROBLEM - Disk space on lead is CRITICAL: DISK CRITICAL - /var/spool/exim4/db is not accessible: Permission denied [03:10:37] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Puppet has 1 failures [03:11:57] RECOVERY - DPKG on lead is OK: All packages OK [03:14:37] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [03:17:20] (03PS1) 10Faidon Liambotis: spamassassin: don't create user/group debian-spamd [operations/puppet] - 10https://gerrit.wikimedia.org/r/143555 [03:17:43] (03CR) 10Faidon Liambotis: [C: 032] spamassassin: don't create user/group debian-spamd [operations/puppet] - 10https://gerrit.wikimedia.org/r/143555 (owner: 10Faidon Liambotis) [03:18:21] (03CR) 10Faidon Liambotis: [V: 032] spamassassin: don't create user/group debian-spamd [operations/puppet] - 10https://gerrit.wikimedia.org/r/143555 (owner: 10Faidon Liambotis) [03:20:20] (03PS1) 10Faidon Liambotis: spamassassin: fix failed dependency [operations/puppet] - 10https://gerrit.wikimedia.org/r/143556 [03:20:39] (03CR) 10Faidon Liambotis: [C: 032 V: 032] spamassassin: fix failed dependency [operations/puppet] - 10https://gerrit.wikimedia.org/r/143556 (owner: 10Faidon Liambotis) [03:21:40] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Complete puppet failure [03:21:50] PROBLEM - spamassassin on lead is CRITICAL: PROCS CRITICAL: 0 processes with args spamd [03:22:40] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:22:50] RECOVERY - spamassassin on lead is OK: PROCS OK: 3 processes with args spamd [03:24:29] (03PS1) 10Faidon Liambotis: Add AAAA for lead.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/143557 [03:25:03] (03CR) 10Faidon Liambotis: [C: 032] Add AAAA for lead.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/143557 (owner: 10Faidon Liambotis) [03:26:30] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 999.082192522 [03:31:06] (03PS1) 10Faidon Liambotis: mail: add lead as second smarthost, remove mchenry [operations/puppet] - 10https://gerrit.wikimedia.org/r/143558 [03:34:16] (03PS1) 10Faidon Liambotis: MX switch, part 3 [operations/dns] - 10https://gerrit.wikimedia.org/r/143559 [03:34:18] (03PS1) 10Faidon Liambotis: MX switch, part 4 [operations/dns] - 10https://gerrit.wikimedia.org/r/143560 [03:56:14] (03CR) 10KartikMistry: [C: 031] Fix string interpolation in cxserver [operations/puppet] - 10https://gerrit.wikimedia.org/r/143361 (owner: 10Nikerabbit) [03:59:22] (03CR) 10KartikMistry: [C: 031] "LGTM as it is fixed as per long discussion and agreement :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140723 (owner: 10Nikerabbit) [04:40:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:38:42 UTC [04:42:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:38:42 UTC [04:44:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:38:42 UTC [04:46:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:38:42 UTC [04:48:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:38:42 UTC [04:49:15] odd [04:50:33] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Wed Jul 2 04:50:25 UTC 2014 [04:51:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:50:25 UTC [04:53:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:50:25 UTC [04:55:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:50:25 UTC [04:57:53] PROBLEM - Puppet freshness on db1058 is CRITICAL: Last successful Puppet run was Wed 02 Jul 2014 04:50:25 UTC [04:58:10] icinga: Warning: The results of service 'Puppet freshness' on host 'db1058' are stale by 0d 0h 0m 45s (threshold=0d 0h 1m 15s) [04:58:37] what breaks a threshold check like that [04:58:43] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Wed Jul 2 04:58:41 UTC 2014 [06:27:31] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:30] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:30] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 2 failures [06:28:30] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:40] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:40] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:40] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 2 failures [06:28:40] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:41] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:41] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 2 failures [06:28:41] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:50] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:50] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:50] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:50] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 2 failures [06:28:50] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:00] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:00] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:10] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:20] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:20] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:20] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:40] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:50] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:20] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:40] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.023 second response time [06:36:50] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:37:40] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [06:45:01] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:45:11] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:45:21] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:45:31] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:45:41] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:45:41] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:45:41] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:45:51] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:46:11] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:46:41] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:46:41] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:46:41] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:46:42] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:46:42] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:52] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:52] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:52] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:46:52] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:47:01] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:47:01] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:47:21] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:47:41] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:50:53] <_joe_> we don't know about service dependencies do we [07:01:32] (03Abandoned) 10Matanya: apt::puppet: remove puppet 2.7 conditional [operations/puppet] - 10https://gerrit.wikimedia.org/r/143283 (owner: 10Matanya) [07:02:16] (03CR) 10Matanya: [C: 031] Remove unused upstream Openstack module [operations/puppet] - 10https://gerrit.wikimedia.org/r/141835 (owner: 10Andrew Bogott) [07:35:46] good morning [07:36:04] hey hashar [07:36:28] godog: I completely forgot about my puppet change for Zuul :-( [07:36:46] hashar: no worries [07:40:02] <_joe_> godog: seee ^, don't you think we need to start to define some service dependency in icinga? [07:40:12] <_joe_> like, if the puppet master fails, all nodes will fail [07:40:41] <_joe_> and we know that, and we don't need to be flooded [07:42:21] +1 [07:42:38] though maintaining dependencies is not an easy task :-/ [07:43:17] indeed, at least for checks that we know for sure are going to fail/flood that'd be good [07:45:24] !log umounted (empty and broken) sdk1 from ms-be3003 and wipe its first sectors, no more remounts [07:45:30] Logged the message, Master [07:47:56] (03PS2) 10Filippo Giunchedi: swift: add icehouse pinning to ms-be1* [operations/puppet] - 10https://gerrit.wikimedia.org/r/143282 [07:48:03] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: add icehouse pinning to ms-be1* [operations/puppet] - 10https://gerrit.wikimedia.org/r/143282 (owner: 10Filippo Giunchedi) [07:49:06] (03CR) 10Hashar: [C: 032] "The change only impacts -labs file which is safe. The service fully rely on beta and not on production at all or another labs project." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140723 (owner: 10Nikerabbit) [07:49:20] (03Merged) 10jenkins-bot: Enable ContentTranslation extension on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140723 (owner: 10Nikerabbit) [07:49:55] PROBLEM - puppet last run on ms-be3003 is CRITICAL: CRITICAL: Puppet has 1 failures [07:54:46] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.001 second response time [07:58:45] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 1 failures [07:59:10] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: Fetching readonly [08:00:31] !log upgrading ms-be1001 to swift icehouse [08:00:36] Logged the message, Master [08:08:12] <_joe_> we need one more puppetmaster i'd say [08:14:50] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 6.878 second response time [08:16:44] ah, strontium/palladium are at capacity? [08:17:50] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [08:18:01] <_joe_> godog: they contionue to spawn 502s from time to time [08:18:09] <_joe_> gotta love puppet scalability [08:18:38] <_joe_> maybe we can look in optimizing them if it hasn't been done thoroughly [08:20:15] hashar: thanks. [08:20:51] hashar: https://gerrit.wikimedia.org/r/#/c/143361/ if you're free :) [08:21:07] ahah [08:21:09] usual typo [08:21:20] (03CR) 10Hashar: "usual typo :-D" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143361 (owner: 10Nikerabbit) [08:21:27] (03CR) 10Hashar: [C: 031] Fix string interpolation in cxserver [operations/puppet] - 10https://gerrit.wikimedia.org/r/143361 (owner: 10Nikerabbit) [08:21:43] akosiaris: would you merge a trivial puppet typo for cxserver please ? https://gerrit.wikimedia.org/r/#/c/143361/ :-D [08:26:26] PROBLEM - MySQL Processlist on db1068 is CRITICAL: CRIT 73 unauthenticated, 0 locked, 0 copy to table, 0 statistics [08:27:26] RECOVERY - MySQL Processlist on db1068 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [08:29:46] (03CR) 10Alexandros Kosiaris: [C: 032] Fix string interpolation in cxserver [operations/puppet] - 10https://gerrit.wikimedia.org/r/143361 (owner: 10Nikerabbit) [08:32:47] hashar: done [08:36:37] kart_: merged :D [09:04:01] (03PS2) 10Matanya: swift: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/140654 [09:04:49] godog: can you please review this before i need to do another painful rebase ? :) [09:05:07] hashar: akosiaris: \0/ [09:05:17] (03CR) 10jenkins-bot: [V: 04-1] swift: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/140654 (owner: 10Matanya) [09:05:55] matanya: seems like jenkins does not like it [09:06:05] me too :/ [09:06:27] :D [09:08:07] matanya: thanks! can it way until I'm done with https://wikitech.wikimedia.org/wiki/Swift/Icehouse however? [09:08:12] wait even [09:08:38] as long as i keep rebasing godog:) [09:09:34] matanya: rebasing swift you mean? I don't expect many changes soon hence painless rebases in theory [09:09:59] matanya: btw I wasn't aware you were on a quest to remove tabs from puppet :) [09:10:09] then cool, i thought that about this one too, but you see what happened ... [09:10:28] godog: my goal is to allow jenkins to vote on style [09:11:32] heheh [09:16:29] (03PS3) 10Matanya: swift: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/140654 [09:18:08] godog: at your spare time, it is ready. I'll keep it rebased as needed. thanks! [09:18:56] matanya: ack, I'll eyeball it but the timing is a bit unfortunate :( thanks for keeping it up to date :)) [09:21:07] (03PS2) 10Matanya: cache: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/140678 [09:21:46] (03PS2) 10Matanya: mailrelay: convert 'true' into a real boolean [operations/puppet] - 10https://gerrit.wikimedia.org/r/143251 [09:22:00] (03PS2) 10Matanya: apt: minor lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/143271 [09:22:28] * matanya keeps rebasing forever ... [09:22:42] godog: I am running those 3 through the catalog differ, I have them [09:22:48] plus some others :-) [09:23:29] matanya: I fixed it yesterday, it should not be long now [09:23:38] thanks akosiaris! [09:28:48]