[00:46:54] Syntax error on line 45 of /etc/apache2/sites-enabled/bugzilla.wikimedia.org: [00:46:54] SSLCertificateFile: file '/etc/ssl/certs/star.wikimedia.org.crt' does not exist or is empty [00:46:55] ...fail! [00:48:00] those certificates are meant to come from puppet, I think [00:49:39] install_certificate { "star-wikimedia": host => "bugzilla.wikimedia.org" } IIRC [00:55:34] i wonder if someone changed or removed that recently, heh [01:05:16] binasher: is that a precise host ? [01:05:30] oh nm [01:06:09] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:16:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [01:30:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:42:08] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 261 seconds [01:42:35] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 252 seconds [01:48:35] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 611s [01:50:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:44] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 25 seconds [01:53:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.391 seconds [01:55:29] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 10s [01:56:23] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 2 seconds [02:20:09] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [02:26:09] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [02:28:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.016 seconds [02:44:37] RECOVERY - Puppet freshness on virt1002 is OK: puppet ran at Wed Jul 25 02:44:12 UTC 2012 [02:54:30] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Wed Jul 25 02:54:12 UTC 2012 [04:03:59] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [04:06:59] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [04:41:02] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [06:49:29] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/15963 [06:51:33] Change merged: Tim Starling; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15440 [07:35:16] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [08:34:32] morning [08:40:33] PROBLEM - Puppet freshness on srv198 is CRITICAL: Puppet has not run in the last 10 hours [08:47:27] PROBLEM - Puppet freshness on potassium is CRITICAL: Puppet has not run in the last 10 hours [09:04:31] * apergos checks the timestamp [09:04:34] yes, it was :-P [09:05:31] hahaha [09:24:06] hell o:-) [09:24:14] damn its is 11:30 already [09:24:52] paravoid: do you want to do the NFS beta stuff ? [09:24:59] I would have to move out in 45 minutes though [09:25:44] should be 5' thing, shouldn't it? :) [09:26:56] should indeed :-D [09:27:28] change is https://gerrit.wikimedia.org/r/#/c/15545/ [09:27:41] I did test it in labs [09:27:45] though not in production :D [09:27:59] ah, needs rebase [09:28:04] should I try the rebase button? :) [09:28:20] although it says "merge or rebase locally" [09:28:41] if there is a conflict that will not work [09:29:08] The rebase failed since conflicts occured during the merge. [09:29:09] yay [09:29:14] care to rebase it? [09:29:22] doing it right now [09:29:48] grmblblb [09:30:06] <<<<<<< HEAD [09:30:06] ======= [09:30:08] all my code [09:30:11] >>>>> my change [09:30:55] ah that is role::applicationserver::labs that got moved [09:32:26] someone killed my code apparently :/ [09:33:50] paravoid: notpeter cleaned up the apache files to get ride of the ::labs role :-D [09:33:57] using if $::realm instead hehe [09:34:18] oh dear... [09:35:41] if $lvsrealserver == true { [09:35:42] include nfs::upload } [09:35:43] kind of strange [09:36:16] which also mean the jobrunner do not have ffs::upload anymore [09:36:21] grr nfs::upload [09:36:58] okay, let me grab a coffee while you're at it [09:37:06] ("should be a 5' thing", famous last words) [09:39:57] indeed [09:40:04] New patchset: Hashar; "include nfs::upload if $upload, not $lvsrealserver" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16611 [09:40:22] paravoid: ^^^ apparently wrong parameter was used [09:40:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16611 [09:40:45] well then rebase my change on that one [09:40:56] and edit the condition to include the incoming nfs::upload::labs [09:51:53] New patchset: Hashar; "apaches::monitoring::labs wraps to apaches::monitoring" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16612 [09:52:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16612 [09:53:41] New patchset: Hashar; "(bug 38084) uses /data/project instead of NFS instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15545 [09:53:47] and rebased my change on top of it [09:54:02] paravoid: so you got to review https://gerrit.wikimedia.org/r/16611 https://gerrit.wikimedia.org/r/16612 which are prerequisites [09:54:05] :-D [09:54:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15545 [09:55:00] yeah, on it [09:57:33] gah, I hate this [09:57:45] mark: you were talking with notpeter about apache.pp, didn't you? [09:58:18] both approach are valid, I guess that is a matter of agreed on a coding style [09:58:30] who's idea was the "if $apache == true { include apache }"? [09:58:32] weren't ops willing to write such a guide? [09:59:57] puppet is a declarative language [10:00:04] if/elses should be used seldomly [10:01:17] New review: Faidon; "I hate the rest of it, but the change itself is sane" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/16611 [10:01:18] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16611 [10:03:14] New review: Faidon; "Labs != beta, but oh well..." [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/16612 [10:03:14] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16612 [10:03:39] oh dear [10:08:36] mark: also, parameterized role classes? doesn't that defeat the point of role classes? [10:08:59] isn't that just the common one ? [10:09:11] could be made private or something ;-D [10:09:23] ? [10:09:56] oh dear, I just saw that [10:09:59] oh dear god [10:10:16] why, oh why [10:13:39] hashar: so, beta nfs? [10:14:04] https://gerrit.wikimedia.org/r/15545 [10:14:05] rebased it [10:14:11] but need to get out to lunch with my wife :/ [10:14:17] well :-] [10:14:24] but :-/ about getting out right now [10:14:43] feel free to apply it if it is good [10:14:48] I will fix the labs post lunch [10:15:21] (I will just have to stop squid, run rsync again, umount stuff, rerun puppet to have the sym links set) [10:16:31] I can wait :) [10:17:15] should be back in an hour and a half roughly :-D Will ping you! [10:17:19] out for now :-] [10:18:19] does this thing even work?!?! [10:20:32] hey [10:20:40] what are you talking about? [10:22:11] manifests/role/apache.pp [10:22:46] what about it? [10:23:05] the tons of if/elses [10:23:21] in a common class, that then gets included parameterized below [10:23:28] yeah I commented about that [10:23:32] oh you did? [10:23:33] not sure if peter has changed that yet [10:23:36] apparently not [10:23:47] https://gerrit.wikimedia.org/r/#/c/16122/ [10:23:59] i said i'd sit with him after he's changed this initial stuff [10:24:28] other than the silly stuff, parameters are fine for a common class that's included by the actual role classes [10:24:41] ah, why didn't we merge that? [10:24:44] it's just comments, isn't it? :) [10:25:20] almost just comments [10:25:42] I don't see the point of doing class common($foo=false) { if $foo { } } class foo { class { "common": foo => true } } [10:25:46] I think it's very bad style [10:25:55] that's what I said isn't it [10:25:59] yes [10:26:06] I'm not arguing with you, just agreeing :) [10:26:09] there's no point in putting everything in a common class when it's not common at all [10:26:15] an "include" works just as well [10:26:41] I feel a lot better after seeing your comments [10:26:44] hehe [10:27:09] there will be more comments, but i'll let him fix this first [10:27:13] I thought this was after your reviews [10:27:19] nah I hadn't seen any of this [10:27:58] btw, your gerrit commit is a total fail of our code-review [10:28:25] the fact that you had to do # FIXME: instead of commenting on the code review system [10:28:37] aanyway [10:28:38] i know [10:28:46] for a long time I tried to review everything going in [10:28:48] but only after merge [10:28:51] and that just doesn't work :( [10:29:03] yeah, I'd like seeing all the commits too [10:29:20] not necessarily review them [10:29:27] yeah [10:29:33] but i've given up after a while [10:29:36] too frustrating, the process :( [10:29:47] yeah [10:29:56] do you read wikitech? :) [10:29:58] no followup process [10:29:58] no [10:30:20] there's a discussion about a gerrit replacement [10:30:28] along with this: http://www.mediawiki.org/wiki/Git/Gerrit_evaluation [10:30:36] i've seen that [10:30:53] I've stepped in a bit, mostly to talk about the lack of a post-merge workflow with gerrit [10:31:18] someone even made stats, something like 80% of commits in ops/puppet are self-reviewed [10:32:07] Ryan is a big fan though :) [10:33:02] hmm [10:33:04] ryan fans [10:33:24] I have a question [10:33:43] how much punsishment would commons be able to take in terms of number of edits per minute [10:33:55] has there been any research on this? [10:34:23] I am not trying to break the site but we are discussing how much trrotelling of the bots are needed [10:34:32] *hrotelling [10:34:54] I cant type :( [10:52:15] paravoid: apparently this is peter's response/followup work: https://gerrit.wikimedia.org/r/#/c/16532/3 [10:52:20] haven't looked at it yet, will do so in a bit [11:04:44] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 8.61962793651 (gt 8.0) [11:07:08] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 0.81811864 [11:07:35] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [11:31:25] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:39:53] New review: Mark Bergsma; "More comments! Inline." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/16532 [11:46:30] and I am back paravoid [11:50:40] and I'm about to leave :) [11:52:21] :-( [11:52:28] will context switch to something else [11:52:38] paravoid: will you be there later this afternoon? [11:56:53] yes [11:56:58] I'm just going out for lunch [11:57:17] after three of four days at home, I need to see a bit of a sun :) [11:59:22] yeah [11:59:26] i'm working outside [11:59:47] it's lovely [12:00:53] it's peaking at 36 today and 37 tomorrow [12:00:57] so, I don't want to do that :) [12:01:20] anyway, bye for now [12:07:21] just 31 here [12:07:24] i'm sitting in the shade ;) [12:07:26] cya [12:33:53] PROBLEM - Puppet freshness on cp1020 is CRITICAL: Puppet has not run in the last 10 hours [12:34:56] PROBLEM - Puppet freshness on mw58 is CRITICAL: Puppet has not run in the last 10 hours [12:35:59] PROBLEM - Puppet freshness on srv209 is CRITICAL: Puppet has not run in the last 10 hours [12:36:53] PROBLEM - Puppet freshness on db35 is CRITICAL: Puppet has not run in the last 10 hours [12:37:56] PROBLEM - Puppet freshness on lvs5 is CRITICAL: Puppet has not run in the last 10 hours [13:11:28] New patchset: Hashar; "manpages for our misc scripts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16606 [13:11:45] and I though it was a draft [13:11:46] bah [13:12:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16606 [13:12:42] <^demon> Drafts will show up on IRC anyway. [13:13:07] New patchset: Bhartshorne; "updating eqiad cluster hash. adding ms-be1009-12 to dhcp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16626 [13:13:44] New review: Hashar; "Still need to complete the over manpages and add a puppet class to install the script in /usr/local/..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/16606 [13:13:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16626 [13:14:09] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16626 [13:14:57] why are you located ben? [13:15:01] where [13:15:08] you're up too early for the west coast ;) [13:15:55] I'm in Massachussetts atm. [13:15:57] New review: Hashar; "over -> other." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/16606 [13:16:10] ah :) [13:16:27] paravoid: can i merge your apache monitoring changces on sockpuppet? [13:16:44] maplebed: hello ben :-) [13:16:53] (I've been on the east coast since wikimania) [13:16:57] maplebed: we did some cleanup this morning they are probably safe [13:17:07] probably doesn't make me very happy... [13:17:10] paravoid is out for lunch / get fresh air. [13:17:22] it means that when I merge it I have to watch it. [13:17:31] but I'm kinda in the middle of something else. [13:17:38] oh [13:17:46] you can revert his change I guess [13:18:02] :-( [13:18:21] lemme look at it again to see what I'd have to watch. [13:18:35] I have no idea what got merged and what was not. [13:18:57] https://gerrit.wikimedia.org/r/#/c/16612/ (commit 6ece81f965 ) is a wrapper for a deprecated class [13:19:12] only going to cause troubles on labs (and I am not even sure labs actually uses that class anyway :-D [13:19:20] I think it's two changes [13:19:29] https://gerrit.wikimedia.org/r/#/c/16611/ and https://gerrit.wikimedia.org/r/#/c/16612/ [13:20:05] ah 16611 is some cleanup $lvsrealserver was used to conditionally include nfs::upload [13:20:25] which would mean that any roles not having lvsrealserver set to true would lack nfs::upload :-) [13:20:32] and bits apaches probably received /mnt/upload too [13:20:41] might be a bit more tricky to watch / verify [13:20:58] well, because of how puppet works it won't unmount upload on those hosts, [13:21:05] but it does mean that new hosts won't have ti. [13:21:06] it. [13:21:38] mark: do you know if it will cause an error to have a conditional with no body in puppet? [13:21:50] not sure [13:21:50] i.e. if $var == true {} [13:22:03] why would you ever have that though? [13:22:12] I'm just looking at the change. [13:22:16] odd [13:22:21] it stripped out the content but left the conditional. [13:22:24] https://gerrit.wikimedia.org/r/#/c/16611/1/manifests/role/apache.pp [13:22:48] that's not an empty body [13:22:56] it's not "include" means: "insert the contents of that class here" [13:22:59] <^demon> maplebed: How long are you in Boston? [13:23:17] ^demon: on cape cod actually. south yarmouth. [13:23:45] <^demon> Ah, I'm going to be in Boston Aug 18-19 visiting some family friends I haven't seen in years. Just wondering :) [13:24:10] I'm flying home on the 29th of july. no overlap. [13:25:27] I do love east coast storms though. Yesterday afternoon: http://screencast.com/t/SqhvcIIbk [13:26:06] <^demon> Yeah, that was yesterday afternoon here too. [13:26:21] <^demon> Blazingly hot all day, clouds rolled in around 3ish, stormed til about 8. [13:26:59] hashar: I'm going to revert https://gerrit.wikimedia.org/r/#/c/16611/1/manifests/role/apache.pp - both because it seems like a change that affects many servers and because it leaves a conditional with no body. [13:27:12] hashar: https://gerrit.wikimedia.org/r/#/c/16612/1/manifests/apaches.pp seems harmless (outside of labs) though, so I'll merge that one. [13:27:22] are you ok with that? [13:27:42] PROBLEM - Host srv206 is DOWN: PING CRITICAL - Packet loss = 100% [13:27:45] maplebed: I'm re-implementing a lot of that in further apache cleanup [13:28:01] hashar: if this is an issue, ping me [13:28:10] as I'm going to do a lot more on those manifests in the next couple of days [13:28:13] mark: I was looking at the $lvsrealserver conditional. [13:28:26] after removing the include, all it has is a comment. [13:28:38] yeah [13:28:39] true [13:28:45] odd change [13:29:06] er? [13:29:12] huh [13:29:35] maplebed: cleanup continues on those manifests. if you need to revert, go for it [13:29:46] k. [13:29:50] New patchset: Bhartshorne; "Revert "include nfs::upload if $upload, not $lvsrealserver"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16627 [13:30:04] that is an arifact of a copy/paste error of mine [13:30:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16627 [13:30:33] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16627 [13:30:42] paravoid: ^^^ [13:31:20] hooray for reading sockpuppet's fetch diff. :P [13:33:41] maplebed: sorry had some issue locally [13:34:01] maplebed: fine with me :-] will have faidon to reapply 16611 (the empty body one) [13:35:07] mark: do you have time to poke at the switch for ms-be1009-12 or should I wait for leslie? [13:35:29] I think networking is not yet set up for those hosts. [13:35:29] what's the issue? [13:35:32] ok [13:35:42] I don't have port numbers, [13:35:55] but they should have all attempted to PXE boot in the last 5 minutes or so [13:36:23] didn't leslie set up some? [13:36:30] i see none on the row C switch [13:36:47] she tried yesterday for ms-be1005-8 but it didn't work. [13:37:03] what was the problem there? [13:37:32] she could probably explain it better... I think she couldn't find the mac addresses on the switch [13:37:41] do you have the macs? [13:37:48] i do. I'll pm them to you. [13:37:52] ok [13:38:01] hashar: I think that I've done a lot of the stuff that you were doing I've done in https://gerrit.wikimedia.org/r/#/c/16532/ [13:38:12] which I'm continuing to work on [13:38:44] notpeter: ahh good to know :-) [13:38:49] 1005-8 are in C2, 1009-12 are in C3. [13:39:51] heh [13:39:56] that would be because the respective switch ports are disabled [13:40:04] then the switch can't learn any mac addresses either [13:40:51] ossm. [13:41:45] also, wow. that was one of the worst sentences I've written in a long time... [13:42:09] notpeter: what, only two extra words... [13:42:21] just take those out and you're fine. [13:43:12] I need to use | grep --no-stupid [13:43:16] on all of my IMs [13:43:17] New review: Nikerabbit; "Scheduling this for i18n deployment next Tuesday." [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/16498 [13:44:50] notpeter: I will also have faidon to apply https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=commitdiff;h=2307433e4ac0a882d035287f204565f1c810390f;hp=6ece81f96556979ac9eae1a34156e0efc0b66176 [13:45:12] notpeter: that introduce the new nfs::upload::labs and use that when realm is not production. [13:45:50] notpeter: the reason is that the beta cluster will just uses symbolic links to some subdirectories of /data/project (a per project shared directory) [13:48:34] hashar: cool, that looks good [13:48:50] I was trying to figure out exactly what to do with nfs::upload::labs [13:48:55] it's already in apaches.pp [13:49:08] mar_k asked me to move it over, which seemed reasonable [13:49:14] so yeah, that stuff looks excellent [13:49:48] well I am not sure how we would prefer to handle it [13:49:54] s/we/ops/ [13:50:10] I know Faidon is not a fan of if( $realm ) {} else {} [13:50:30] it's not ideal, but it's how we're doing things currently [13:50:36] I'm open to other suggestions [13:51:09] I am fine with any way as long as I can easily hack the puppet files to insert my 'beta' specific kind :-) [13:51:37] I for sure love the parameterized common class [14:03:54] New patchset: Mark Bergsma; "Add public1-c-eqiad and private1-c-eqiad subnets" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16629 [14:04:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16629 [14:05:00] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16629 [14:05:12] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [14:05:51] New patchset: Hashar; "Fix bits VLC when enable_geoiplookup is disabled" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15445 [14:06:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15445 [14:06:31] New review: Hashar; "Patchset 5 pass the hostname (test.wikipedia.org) and escape it in the bits.inc.vcl.erb template." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15445 [14:09:24] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [14:16:35] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [14:17:11] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [14:17:19] New review: Hashar; "Patchset 12 rely on cluster_options hash that was introduced in I033e5b7f and set the default values..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [14:27:55] New review: Hashar; "Patchset 12 was deployed on deployment-cache-bits02. See http://commons.wikimedia.beta.wmflabs.org/w..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [14:28:05] New review: Hashar; "Patchset 5 was deployed on deployment-cache-bits02. See http://commons.wikimedia.beta.wmflabs.org/wi..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15445 [14:37:32] Logged the message, Master [14:41:39] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [14:52:36] RECOVERY - swift-object-replicator on ms-be1002 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-replicator [14:52:54] RECOVERY - swift-container-replicator on ms-be1002 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-replicator [14:58:45] RECOVERY - swift-container-replicator on ms-be1001 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-replicator [14:58:54] RECOVERY - swift-object-replicator on ms-be1001 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-replicator [14:59:39] RECOVERY - swift-account-replicator on ms-be1002 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-replicator [15:11:48] heya [15:11:57] that took longer than expected :) [15:12:20] PROBLEM - swift-container-auditor on ms-be1005 is CRITICAL: Connection refused by host [15:12:21] PROBLEM - swift-container-server on ms-be1006 is CRITICAL: Connection refused by host [15:12:21] PROBLEM - swift-object-server on ms-be1006 is CRITICAL: Connection refused by host [15:12:21] PROBLEM - swift-account-replicator on ms-be1006 is CRITICAL: Connection refused by host [15:12:21] PROBLEM - swift-account-auditor on ms-be1009 is CRITICAL: Connection refused by host [15:12:21] PROBLEM - swift-object-auditor on ms-be1005 is CRITICAL: Connection refused by host [15:12:21] PROBLEM - swift-container-auditor on ms-be1009 is CRITICAL: Connection refused by host [15:12:22] PROBLEM - swift-object-auditor on ms-be1009 is CRITICAL: Connection refused by host [15:12:22] PROBLEM - swift-account-auditor on ms-be1005 is CRITICAL: Connection refused by host [15:12:38] PROBLEM - swift-account-reaper on ms-be1005 is CRITICAL: Connection refused by host [15:12:38] PROBLEM - swift-object-replicator on ms-be1005 is CRITICAL: Connection refused by host [15:12:38] PROBLEM - swift-container-replicator on ms-be1005 is CRITICAL: Connection refused by host [15:12:38] PROBLEM - swift-account-reaper on ms-be1009 is CRITICAL: Connection refused by host [15:12:38] PROBLEM - swift-container-replicator on ms-be1009 is CRITICAL: Connection refused by host [15:12:39] PROBLEM - swift-container-updater on ms-be1006 is CRITICAL: Connection refused by host [15:12:39] PROBLEM - swift-object-updater on ms-be1006 is CRITICAL: Connection refused by host [15:12:40] PROBLEM - swift-object-replicator on ms-be1009 is CRITICAL: Connection refused by host [15:12:40] PROBLEM - swift-account-server on ms-be1006 is CRITICAL: Connection refused by host [15:12:56] PROBLEM - swift-object-auditor on ms-be1006 is CRITICAL: Connection refused by host [15:12:56] PROBLEM - swift-container-auditor on ms-be1006 is CRITICAL: Connection refused by host [15:12:56] PROBLEM - swift-account-replicator on ms-be1005 is CRITICAL: Connection refused by host [15:12:56] PROBLEM - swift-container-server on ms-be1005 is CRITICAL: Connection refused by host [15:12:56] PROBLEM - swift-object-server on ms-be1009 is CRITICAL: Connection refused by host [15:12:57] PROBLEM - swift-object-server on ms-be1005 is CRITICAL: Connection refused by host [15:12:57] PROBLEM - swift-account-auditor on ms-be1006 is CRITICAL: Connection refused by host [15:13:06] PROBLEM - swift-container-server on ms-be1009 is CRITICAL: Connection refused by host [15:13:14] PROBLEM - swift-account-replicator on ms-be1009 is CRITICAL: Connection refused by host [15:13:20] paravoid: now look what you've done [15:13:23] PROBLEM - swift-account-reaper on ms-be1006 is CRITICAL: Connection refused by host [15:13:23] PROBLEM - swift-account-server on ms-be1009 is CRITICAL: Connection refused by host [15:13:23] PROBLEM - swift-object-replicator on ms-be1006 is CRITICAL: Connection refused by host [15:13:23] PROBLEM - swift-container-replicator on ms-be1006 is CRITICAL: Connection refused by host [15:13:23] PROBLEM - swift-object-updater on ms-be1009 is CRITICAL: Connection refused by host [15:13:24] PROBLEM - swift-container-updater on ms-be1009 is CRITICAL: Connection refused by host [15:13:24] PROBLEM - swift-container-updater on ms-be1005 is CRITICAL: Connection refused by host [15:13:32] PROBLEM - swift-account-server on ms-be1005 is CRITICAL: Connection refused by host [15:13:32] PROBLEM - swift-object-updater on ms-be1005 is CRITICAL: Connection refused by host [15:14:30] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16420 [15:15:47] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16457 [15:20:30] New review: Reedy; "Where's the talk NS?" [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/16432 [15:22:14] paravoid: so not peter is refactoring apache roles :- [15:22:43] paravoid: and one change got reverted :/ [15:22:55] apparently you forgot to merge it on sock puppet! [15:23:09] Iefedf2b7 Revert "include nfs::upload if $upload, not $lvsrealserver" (MERGED) [15:25:20] New patchset: Hashar; "include nfs::upload if $upload, not $lvsrealserver" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16632 [15:25:59] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16558 [15:25:59] New review: Hashar; "Ben said:" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/16632 [15:25:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16632 [15:26:04] New review: Hashar; "Resubmitted with https://gerrit.wikimedia.org/r/#/c/16632/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16611 [15:27:07] New patchset: Hashar; "include nfs::upload if $upload, not $lvsrealserver" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16632 [15:27:45] New review: Hashar; "Patchset 2 removes the empty block:" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/16632 [15:27:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16632 [15:28:59] paravoid: notpeter: added you to https://gerrit.wikimedia.org/r/#/c/16632/ which make the apache role to include nfs::upload when $upload is true instead of when $lvsrealserver is true. [15:29:26] PROBLEM - NTP on ms-be1006 is CRITICAL: NTP CRITICAL: No response from NTP server [15:30:11] PROBLEM - NTP on ms-be1009 is CRITICAL: NTP CRITICAL: No response from NTP server [15:30:11] PROBLEM - NTP on ms-be1005 is CRITICAL: NTP CRITICAL: No response from NTP server [15:31:36]