[00:01:26] (03CR) 10Tim Landscheidt: [C: 031] "Tested on tools-exec-05." [operations/puppet] - 10https://gerrit.wikimedia.org/r/100147 (owner: 10Dapete) [02:06:31] !log LocalisationUpdate completed (1.23wmf7) at Mon Dec 30 02:06:31 UTC 2013 [02:06:59] Logged the message, Master [02:11:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [02:21:00] !log LocalisationUpdate completed (1.23wmf8) at Mon Dec 30 02:21:00 UTC 2013 [02:21:18] Logged the message, Master [02:32:44] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 30 02:32:44 UTC 2013 [02:32:59] Logged the message, Master [04:49:41] PROBLEM - Apache HTTP on mw1156 is CRITICAL: Connection timed out [04:49:41] PROBLEM - Apache HTTP on mw1157 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:11] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: Connection timed out [04:50:13] PROBLEM - Apache HTTP on mw1159 is CRITICAL: Connection timed out [04:50:14] PROBLEM - Apache HTTP on mw1160 is CRITICAL: Connection timed out [04:50:31] PROBLEM - Apache HTTP on mw1154 is CRITICAL: Connection timed out [04:50:33] greg-g: look up btw [04:50:41] PROBLEM - Apache HTTP on mw1153 is CRITICAL: Connection timed out [04:50:41] PROBLEM - Apache HTTP on mw1158 is CRITICAL: Connection timed out [04:50:41] PROBLEM - Apache HTTP on mw1155 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:32] RECOVERY - Apache HTTP on mw1157 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.066 second response time [04:51:32] okay [04:51:33] two things [04:51:37] #1 who broke rendering [04:51:53] #2 who pushed production changes on a holiday weekend sunday [04:52:01] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [04:52:07] i imagine i can possibly find out #2 [04:52:11] RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 9.349 second response time [04:52:21] RECOVERY - Apache HTTP on mw1154 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.066 second response time [04:52:28] yeah, #2 was more of a "this is an angry rhetorical question" [04:52:31] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.679 second response time [04:52:38] right :) [04:53:01] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 70390 bytes in 0.199 second response time [04:53:03] RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.059 second response time [04:53:10] i was planning to put in some review reqs first thing tomorrow morning :) [04:53:20] (and bump some from last week) [04:53:31] RECOVERY - Apache HTTP on mw1158 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.058 second response time [04:53:43] :) [04:53:45] cool [04:54:31] RECOVERY - Apache HTTP on mw1153 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.046 second response time [04:55:31] RECOVERY - Apache HTTP on mw1156 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.058 second response time [04:56:34] whoa, https://gdash.wikimedia.org/dashboards/reqerror/ is dramatic [04:56:57] looks recovered though [04:58:00] yeah [04:58:03] so .... yay ? [05:00:08] not yet recovered per the bot: 30 04:52:01 <+icinga-wm> PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [05:01:29] yeah, but gdash looks happy -- i'll stick around for a little while :) [05:01:46] erm, i can't figure out how to access icinga web ui [05:01:57] i even tried disabling HTTPS everywhere [05:02:35] oh, there's a security flaw so we enabled authentication on it [05:02:43] oh, yay [05:02:46] haven't gotten a backported package either made or created yet :-/ [05:02:52] ganglia redux [05:03:04] yep [05:03:08] :( [05:05:18] heh, /me spies a LeslieCarr @ RT 5064 [05:05:38] haha [05:05:40] oh yes [05:05:49] ironic ;) [05:07:09] i'm going to stick around for a few, double check that the errors go down [05:07:33] what the [05:07:46] those are rendering boxes? [05:08:55] greg-g's late to the party. i gave an early invitation! [05:09:04] yeah, well, sunday [05:09:25] site.pp says: # mw1153-1160 are imagescalers (precise) [05:09:31] * greg-g nods [05:09:51] greg-g: right but you were active in #wikimedia so i thought i'd ping you here :) [05:10:02] well, "active" [05:10:25] but yeah, thanks :) [05:10:27] ok, i guess it was mostly Gloria [05:12:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [05:13:57] ok, back to sleep [05:15:19] errors look back to normal [05:21:12] LeslieCarr: I don't see any code updates in SAL (aaron was doing scap testing with nops, but he did make a jobqueue change there recently...) [05:21:52] which was just https://gerrit.wikimedia.org/r/#/c/104480/ [05:56:01] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [06:07:41] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:11:31] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [07:30:50] (03PS2) 10Ori.livneh: Added MW version argument to scap [operations/puppet] - 10https://gerrit.wikimedia.org/r/104189 (owner: 10Aaron Schulz) [07:31:30] (03CR) 10Ori.livneh: [C: 032] Added MW version argument to scap [operations/puppet] - 10https://gerrit.wikimedia.org/r/104189 (owner: 10Aaron Schulz) [08:13:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [08:32:52] I will be afk for a couple of hours (friend here from out of town), reachable by cell if something horrible happens [08:33:00] (03PS1) 10Ori.livneh: Update public domain mark, using text from CC0 [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104491 [08:33:30] * ori waves [08:33:44] (03CR) 10Ori.livneh: [C: 032 V: 032] Update public domain mark, using text from CC0 [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/104491 (owner: 10Ori.livneh) [10:48:08] (03CR) 10Alexandros Kosiaris: [C: 04-2] "LVS servers are known to exhibit big problems when NTP is running on them. The fact that they are not running NTP is by design and not by " [operations/puppet] - 10https://gerrit.wikimedia.org/r/20681 (owner: 10Faidon Liambotis) [11:09:00] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Nitpic. On a side note, everything under the Exec line needs a rewrite. It is all about manipulating a couple of files so IMHO it should b" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104339 (owner: 10Matanya) [11:10:31] akosiaris: have time for a few questions not related to this patch please? [11:10:49] yeah [11:11:50] (03CR) 10Faidon Liambotis: "Yes, past experience with such behavior is the reason this changeset has been sitting for over a year, as is documented in a few places. W" [operations/puppet] - 10https://gerrit.wikimedia.org/r/20681 (owner: 10Faidon Liambotis) [11:12:20] manifests/misc/irc.pp has only one little class used in some places. should it be a module? [11:14:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [11:14:38] akosiaris, also how can i download a patch done by someone else on top of a patch by me? review -d didn't bring those changes [11:15:54] last question: admins.pp really hurts my eyes. what should be the correct approach to handle this? [11:17:28] matanya: on the first question, the correct way would be the trifecta in a single class module and then a role class that includes the monitor_service and that class and that role class should be included in site.pp and wikibugs.pp. It is indeed too small though so it might note make the best of sense. I 'd be for that approach though just for the consistency benefits [11:18:22] on the second question, you just get the patch from gerrit and apply it manually with patch. It can not be done with git review AFAIK [11:20:04] finally on the last question, a define would certainly help to remove some code duplication [11:21:03] those inheritances then should be removed as well (puppet inheritance sucks) [11:21:22] thanks a lot. i'll fix the braces, and then will go on to the rest [11:23:55] (03PS2) 10Matanya: labs_vmbuilder: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/104339 [11:34:01] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Comments inline." (037 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104348 (owner: 10Matanya) [11:36:14] matanya: .... as far as https://gerrit.wikimedia.org/r/104339, I was not suggesting that approach. I was suggesting [11:36:14] require => [ Package['python-vm-builder'], [11:36:14] File["{blahablah"], [11:36:14] ] [11:36:14] The last line being the important one [11:36:44] the rest was ok. All I wanted was the brace indented some spaces :-) [11:45:45] akosiaris: monday is monday :) [11:46:21] http://en.wikipedia.org/wiki/I_Don't_Like_Mondays ? [11:49:05] (03PS8) 10Matanya: svn: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 [11:51:15] (03PS1) 10Alexandros Kosiaris: Remove etherpad-old [operations/dns] - 10https://gerrit.wikimedia.org/r/104498 [11:53:48] akosiaris: "sometimes it is better to stay in bed on monday in order not to fix monday's work the rest of the week" [11:54:08] hahahaha. so true [11:55:17] should be in our bugzilla quips [11:55:43] (03PS3) 10Matanya: labs_vmbuilder: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/104339 [15:39:01] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [15:39:51] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.41 ms [15:54:41] PROBLEM - NTP on mw31 is CRITICAL: NTP CRITICAL: Offset unknown [15:58:41] RECOVERY - NTP on mw31 is OK: NTP OK: Offset 0.001707911491 secs [16:08:27] thanks akosiaris [16:08:35] :-) [16:10:49] akosiaris: how about a simple planet update? :) https://gerrit.wikimedia.org/r/#/c/103047/ [16:11:58] (03CR) 10Alexandros Kosiaris: [C: 032] [Planet] Update wikimedia.fi URL [operations/puppet] - 10https://gerrit.wikimedia.org/r/103047 (owner: 10Nemo bis) [16:12:16] (03CR) 10Alexandros Kosiaris: [V: 032] [Planet] Update wikimedia.fi URL [operations/puppet] - 10https://gerrit.wikimedia.org/r/103047 (owner: 10Nemo bis) [16:12:27] thanks :D [16:12:35] you are welcome [16:28:37] hiii akosiaris, got any experience with icinga? [16:28:54] ottomata: yeah [16:28:54] i'm trying to figure out why my new nrpe process alerts aren't showing up [16:29:09] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=analytics1021 [16:29:20] I added [16:29:27] nrpe::monitor_service { 'kafka': [16:29:27] nrpe_command => '/usr/lib/nagios/plugins/check_procs -c 1:1 -C java -a "kafka.Kafka /etc/kafka/server.properties"', [16:29:28] a while ago [16:29:36] and i see the checks on neon properly [16:30:02] Error: The description string for service 'Hadoop Namenode (Stand By)' on host 'analytics1009' contains one or more illegal characters. [16:30:02] Error: The description string for service 'Hadoop Namenode (Primary)' on host 'analytics1010' contains one or more illegal characters. [16:30:07] that is your problem [16:30:15] psshhh [16:30:20] it fails to reload its configuration due to those two errors [16:30:24] yay! [16:30:27] you are sucha good log checker [16:30:34] lol [16:30:41] my new role/title [16:30:44] :-) [16:31:01] wait, what log is that in? [16:31:34] no log. Just icinga -v /etc/icinga/icinga.cfg or even simplier [16:31:37] service icinga reload [16:31:39] ohhhhhhh [16:31:41] oh [16:31:47] i hadn't done reload yet because I was afraid of what might happen! [16:31:50] (03CR) 10Chad: [C: 031] "lgtm, will merge at the start of the window." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104134 (owner: 10Manybubbles) [16:32:11] yeah not restart or stop/start sequence [16:32:14] hm ok [16:32:17] or it will not come back again [16:32:20] i guess it doesn't like spaces? [16:32:32] i think it the ( character [16:32:35] or maybe the parens [16:32:36] hgm [16:32:40] but the kafka desc doeesn't have parens [16:33:49] (03PS1) 10Ottomata: Removing parens in nrpe descriptions [operations/puppet] - 10https://gerrit.wikimedia.org/r/104515 [16:34:06] (03CR) 10Ottomata: [C: 032 V: 032] Removing parens in nrpe descriptions [operations/puppet] - 10https://gerrit.wikimedia.org/r/104515 (owner: 10Ottomata) [16:43:23] akosiaris: if I am changing someone's ssh key [16:43:36] and the email/desc for the key hasn't changed [16:43:45] is it better to ensure => absent on the first one and add a new one [16:43:48] or just change the actual key? [16:44:04] ensure => absent [16:44:07] k [16:44:38] i'll have to use a new title then, right? [16:44:38] ssh_authorized_key { 'nuria@wikimedia.org': [16:44:41] maybe [16:44:43] yes [16:44:53] does it have to match what they have in their own key file? [16:44:58] or is it arbitrary? [16:45:02] arbitrary [16:45:04] ok [16:46:41] (03PS1) 10Ottomata: Changing Nuria's ssh key per her request [operations/puppet] - 10https://gerrit.wikimedia.org/r/104521 [16:46:42] (03PS1) 10Ottomata: Changing Dan's production ssh key [operations/puppet] - 10https://gerrit.wikimedia.org/r/104522 [16:47:42] (03CR) 10Ottomata: [C: 032 V: 032] Changing Nuria's ssh key per her request [operations/puppet] - 10https://gerrit.wikimedia.org/r/104521 (owner: 10Ottomata) [16:48:14] (03CR) 10Ottomata: [C: 032 V: 032] Changing Dan's production ssh key [operations/puppet] - 10https://gerrit.wikimedia.org/r/104522 (owner: 10Ottomata) [16:50:14] (03PS1) 10Ottomata: Removing email from Dan's key [operations/puppet] - 10https://gerrit.wikimedia.org/r/104524 [16:50:44] (03CR) 10Ottomata: [C: 032 V: 032] Removing email from Dan's key [operations/puppet] - 10https://gerrit.wikimedia.org/r/104524 (owner: 10Ottomata) [17:01:40] ^d: I'm going to start deploying now. First the elasticsearch side config updates. [17:01:50] <^d> mmk. [17:01:55] <^d> I'm gonna start merging in gerrit [17:01:59] <^d> so jenkins can get going [17:03:21] thanks!@ [17:04:23] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:04:32] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:04:42] RECOVERY - Host virt1005 is UP: PING OK - Packet loss = 0%, RTA = 0.54 ms [17:05:22] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:06:39] ^d: all done with pre updates. moving on to deploy to wmf8 which looks like it merged [17:07:06] <^d> Yep [17:07:12] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Mon Dec 30 17:07:05 UTC 2013 [17:07:13] <^d> Well, we can just scap all at once, right? [17:07:22] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Mon Dec 30 17:07:18 UTC 2013 [17:07:27] <^d> 7's in too [17:07:32] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 05:07:05 PM UTC [17:08:22] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 05:07:18 PM UTC [17:08:30] <^d> manybubbles: Are you wanting to sync config with code, or scap & then sync-file initsettings? [17:08:53] I wanted to do wmf8 first if possible [17:08:58] then wmf7, then config [17:09:20] thats two scaps and a sync-file [17:09:53] I'm actually just now ready for the first scap if that sounds ok [17:11:11] I could do the internationalization updates then the sync-dir [17:11:24] https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Alternative_to_scap [17:11:31] if it is likely to work [17:12:54] greg-g: we've started, btw [17:12:57] what does scap stand for? sync config mumble puppet? [17:13:02] cool [17:13:19] jgage: not sure [17:13:21] <^d> jgage: sync-common-all-php, if memory serves. [17:13:28] ah thanks [17:13:41] ^d: do you have an opinion on which I should do? [17:14:06] <^d> I'd just scap once. [17:14:10] <^d> 7 & 8. [17:14:18] <^d> Then sync config for new wikis with sync-file. [17:14:23] ^d: k. [17:14:25] I'll do that [17:14:40] yaaa, thanks akosiaris, icinga is doing what I want now [17:15:06] <^d> jgage: https://wikitech.wikimedia.org/wiki/Scap [17:16:17] thanks, was just editing to include the definition but perhaps it's not needed [17:16:24] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [17:16:40] ^d: manybubbles@tin:/a/common$ scap 'Update CirrusSearch and Elastica to master' [17:16:42] Invalid MediaWiki version "Update CirrusSearch and Elastica to master" [17:16:47] jgage: no harm [17:17:05] jgage: and then you know if ^d was wrong, someone will fix it (maybe) ;) [17:17:08] <^d> scap syntax changed... [17:17:10] confusingly, higher in that doc it says that sync-common-all just runs scap [17:17:24] jgage: scap also just calls scap-1 [17:17:27] :) [17:17:30] hah great [17:18:47] er, sync-common just calls scap-1 [17:18:53] https://git.wikimedia.org/blob/operations%2Fpuppet.git/df881e08c2f3a7365aa400462e0515206136e75a/files%2Fscap%2Fsync-common [17:18:58] <^d> AaronSchulz: What's the syntax for scap now? [17:19:02] mmm rtfs [17:19:54] ^d: oh, that was merged, it works the same with no arg [17:20:02] ...unless there is a bug :) [17:20:09] <^d> Must be [17:20:15] <^d> How do you sync all versions with a description? [17:20:32] <^d> (See ~15 lines up from manybubbles) [17:20:34] RECOVERY - Puppet freshness on virt1002 is OK: puppet ran at Mon Dec 30 17:20:33 UTC 2013 [17:20:43] Copying to tin from tin.eqiad.wmnet...Unexpected remote arg: tin.eqiad.wmnet::common/ [17:20:56] did $(scap active 'log log log') [17:21:20] <^d> ugh. [17:21:22] it is running though [17:21:24] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 05:20:33 PM UTC [17:21:33] should I shoot it before it does anything? [17:21:55] <^d> Yeah let's stop for a sec. [17:22:04] stopped during [17:22:11] Updating LocalisationCache for 1.23wmf7... ^CUpdate of MediaWiki localisation messages failed [17:23:19] .... [17:23:21] <^d> I'm on tin and going to try this. [17:23:51] * ^d hmms [17:24:17] I'm not seeing ore fatals then normal, so I doubt it caused any real damage [17:24:31] greg-g: yay first deploy in the morning! [17:24:39] happy monday [17:24:40] hmm, 'scap "desc"' won't work though [17:25:00] Aaron|home: $(scap active "desc") does not a big deal [17:25:06] once the docs are updated [17:25:14] !log demon started scap: active Cirrus and Elastica to master [17:25:15] <^d> AaronSchulz: if rsync "${RSYNC_ARGS[@]}" "$SERVER"::common/ "${MW_COMMON}" is wrong I think. [17:25:32] Logged the message, Master [17:25:38] <^d> !log aborted scap [17:25:46] that line is old, all I did was change MW_RSYNC_ARGS to RSYNC_ARGS [17:25:53] ^d: are you saying RSYNC_ARGS is fishy? [17:25:56] Logged the message, Master [17:26:07] <^d> Yep. [17:26:24] <^d> I'm getting mw1097: Unexpected remote arg: mw1070.eqiad.wmnet::common/ consistently. [17:26:29] <^d> Missing a path somewhere? :) [17:26:58] greg-g: does what is being deployed include load balancer class rename? [17:27:05] in core [17:27:25] <^d> The deploy right now? [17:27:27] * aude thinks that is next week or test2 on thursday [17:27:40] ^d: core deploy [17:27:47] <^d> ignore me :) [17:27:51] k [17:28:40] i think the change is backwards compatible fine but is worth paying some extra attention to make sure no problems [17:28:55] ^d: have you tried just "scap"? [17:29:08] !log demon started scap [17:29:16] <^d> !log aborted scap [17:29:20] <^d> Same result. [17:29:25] Logged the message, Master [17:29:39] Logged the message, Master [17:29:56] ^d: all that does is RSYNC_ARGS=MW_RSYNC_ARGS ... how is that even different? [17:30:22] <^d> I don't know. I'm just suspecting that line because of the output I'm getting from tin. [17:30:49] yeah I get that with sync-common [17:33:15] <^d> Hmm. [17:33:28] * Aaron|home is confused [17:34:11] ^d: https://gerrit.wikimedia.org/r/#/c/104189/2/files/scap/scap-2 [17:35:01] <^d> Yeah, I know that what you changed isn't suspect. [17:35:16] I see [17:35:18] no $ [17:35:54] not enough $ [17:36:00] insert more $ [17:36:18] sounds like oracle, actually [17:36:25] <^d> Don't we have a bug for rewriting this in something other than bash? :p [17:36:38] ^d: perl [17:36:54] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Mon Dec 30 17:36:51 UTC 2013 [17:37:14] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 05:36:51 PM UTC [17:37:25] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Mon Dec 30 17:37:21 UTC 2013 [17:37:25] that's not enough though [17:37:34] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 05:37:21 PM UTC [17:37:48] RSYNC_ARGS=${MW_RSYNC_ARGS[@]} [17:37:53] ok, there we go [17:38:04] ^d: bash is by far the worse language I've seen [17:38:18] <^d> I'd vote for redoing it in python. [17:38:20] <^d> :) [17:38:28] Aaron|home: lolcats is at least as bad [17:38:30] manybubbles: Ops prefer things to be in ruby [17:38:36] <^d> node.js! [17:38:40] java [17:38:46] oooh, scala [17:38:54] ok should I scap now? [17:38:56] <^d> coldfusion? [17:39:02] VB .NET [17:39:08] I once went to a coldfusion conference [17:39:16] <^d> Why? [17:39:16] VB.NET is not as shitty as you'd imagine it to be [17:39:27] Is someone messing with virt100x today, or are they just spontaneously unstable? [17:39:28] <^d> Aaron|home: Patch incoming? [17:39:43] manybubbles: Yes and no. I've written enough of it though... [17:39:44] it was only 45% cold fusion. it advertised itself (in some circles) as 10% cold fusion. [17:39:53] Reedy: that was my first job! [17:39:56] Bad venn diagrams [17:40:03] Depends on the overlap [17:40:12] so long as it isn't vb for applications [17:40:16] yeah, ugh, fucking git error [17:40:19] that actually might be the worst language ever [17:40:37] because it has to run inside excel or something. fun to watch though. [17:41:01] You can write standalone apps in VB .NET and such... [17:41:09] It's vbscript in excel? [17:41:23] Reedy: sure. vb.net is a first class peer with C# [17:41:36] it is vbscript in excel. or rather, it is kinda vbscript [17:41:44] <^d> manybubbles: The time tracking application at $DAY_JOBS[0] was one of those "MS Access" macro-applications, originally on top of an Access database. They later upgraded it to use SQL Server. [17:41:54] but you move the curser around and make changes so they have to keep their hands off the keyboard while it works [17:42:11] (03PS1) 10Aaron Schulz: Fixed RSYNC_ARGS assignment in scap-2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104529 [17:42:24] I like C# [17:42:27] ^d: honestly, I don't hate SQL Server that much. I mean, they have balls naming it SQL Server and all. [17:42:28] VB tends to be too wordy though [17:42:35] but beyond that it is reasonably fully featured [17:42:36] (03CR) 10Chad: [C: 031] "scap is broken, please merge." [operations/puppet] - 10https://gerrit.wikimedia.org/r/104529 (owner: 10Aaron Schulz) [17:42:39] C# = Microsoft Java [17:42:47] * Aaron|home ducks [17:42:51] C# is pretty much better than java [17:42:58] as a language [17:43:09] as an ecosystem is always the java argument [17:43:13] but I hate them all [17:43:13] Aaron|home: Because it is their rival to java [17:43:13] <^d> manybubbles: I don't hate it either. It was actually an upgrade from an access db :) [17:43:22] Mono! [17:43:31] ;) [17:44:02] * Reedy nudges greg-g [17:44:09] * Reedy whispers it's not a programming language [17:44:16] (03PS1) 10Jgreen: flip DNS for fundraisingdb-read.wmnet from db78 to db1008 [operations/dns] - 10https://gerrit.wikimedia.org/r/104530 [17:44:59] <^d> manybubbles: Maybe we should rewrite Lucene & Elasticsearch in C#. [17:45:05] ori: https://gerrit.wikimedia.org/r/104529 on fix [17:45:06] (03CR) 10Manybubbles: [C: 031] Fixed RSYNC_ARGS assignment in scap-2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104529 (owner: 10Aaron Schulz) [17:45:08] <^d> Because why not? [17:45:33] what's funny is that RSYNC_ARGS=$MW_RSYNC_ARGS result in the value "-a" [17:45:52] Aaron|home: shakes fist at bash [17:46:40] <^d> Aaron|home: ori's idle for 5.5h. Maybe we need to ping someone else :) [17:46:54] great timing [17:47:05] ottomata: what about you? [17:47:08] I think bd is off today [17:47:11] you merge things [17:47:13] apergos? [17:47:16] do y'all need a merge? [17:47:21] yes [17:47:21] <^d> Yessss [17:47:24] Not that I know anything about scap, but… patch looks legit [17:47:25] eh? [17:47:28] <^d> https://gerrit.wikimedia.org/r/104529 [17:47:53] (03CR) 10Andrew Bogott: [C: 032] Fixed RSYNC_ARGS assignment in scap-2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104529 (owner: 10Aaron Schulz) [17:48:00] ah you took i [17:48:06] t [17:48:21] yep [17:48:31] ah [17:48:46] what aargs were not getting passed? [17:49:30] um… I was pretty much just taking ^d's word for this [17:49:37] asking ^d [17:49:39] that and being well aware that bash never works the first time [17:49:39] <^d> Take Aaron|home's word for it. [17:49:45] oops [17:49:54] <^d> This is the second time ;-) [17:50:14] RECOVERY - Puppet freshness on virt1002 is OK: puppet ran at Mon Dec 30 17:50:10 UTC 2013 [17:50:17] oh I guarantee there is a big difference (between before and after), I just was wondering what it ws being called with [17:50:24] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 05:50:10 PM UTC [17:50:30] apergos: at least -a [17:50:34] I'm not sure beyond that [17:50:42] I'm actually not brain fried at this hour so I can even remember bash syntax, how nice [17:50:56] <^d> andrewbogott: We'll have to force a puppet run on tin. [17:50:57] usually by the time the evening rolls around I would have to test by hand or give up [17:51:05] apergos, I'm going to lunch but will look at those puppet warnings when I return. I suspect that someone (mhoover?) is messing with the servers and happens on to be on irc :( [17:51:06] Remember? What's that? [17:51:12] ^d ok [17:51:20] thanks [17:51:28] I know the virts were in process [17:51:41] so I was not too worried about it yet [17:51:43] yeah, 1007 has been complaining forever because it's broken in some interesting way [17:51:54] Yeah, safe for you to ignore them for the next couple of weeks [17:52:29] at he point where they have accounts on them it would be nice if puppet would run on them [17:52:59] until then I'm less concerned [17:53:39] 'accounts' like, shell accounts? Because they do have shell accounts for mike. but no one else. [17:53:50] ^d: I imagine I should wait twenty minutes or so for scap to be updated [17:54:08] <^d> No, should be up as soon as andrewbogott says puppet is done :) [17:54:09] (03CR) 10Jgreen: [C: 032 V: 031] flip DNS for fundraisingdb-read.wmnet from db78 to db1008 [operations/dns] - 10https://gerrit.wikimedia.org/r/104530 (owner: 10Jgreen) [17:54:15] tin is doing a puppet run right now, will be done in a minute or two [17:54:29] andrewbogott: thanks! [17:54:46] ok, done. And, -> lunch [17:54:51] thanks [17:54:53] starting again [17:55:02] !log manybubbles started scap: active Update CirrusSearch and Elastica to master [17:55:06] <^d> Aaron|home: Can you send an e-mail to engineering about the scap syntax change btw? [17:55:20] Logged the message, Master [17:55:20] mw1049: rsync: -a --delete-delay --delay-updates --compress --delete --exclude=**/.svn/lock --exclude=**/.git/objects --exclude=**/.git/**/objects --exclude=**/cache/l10n/*.cdb --no-perms: unknown option [17:55:22] mw1030: rsync error: syntax or usage error (code 1) at main.c(1453) [client=3.0.9] [17:55:43] mw1072: rsync: -a --delete-delay --delay-updates --compress --delete --exclude=**/.svn/lock --exclude=**/.git/objects --exclude=**/.git/**/objects --exclude=**/cache/l10n/*.cdb --no-perms: unknown option [17:55:44] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:56:11] !log aborted scap [17:56:20] I aborted it. it was blowing up [17:56:26] Aaron|home and ^d: sad [17:56:29] Logged the message, Master [17:56:34] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [17:58:12] what the ef [17:58:54] anyway to just revert to a known "good" (for various definitions of good) version of scap? [18:01:07] !log powering down search1005 to swap /dev/sdb [18:01:23] Logged the message, Master [18:01:30] greg-g: at least I found this problem this morning rather than reedy in an hour [18:01:57] sync-common still giving trouble, even though the only difference is RSYNC_ARGS=${MW_RSYNC_ARGS[@]} and RSYNC_ARGS being used in place of MW_RSYNC_ARGS [18:02:06] since I have no $1 [18:02:23] it's amazing that that could cause it to not work [18:03:14] PROBLEM - Host search1005 is DOWN: PING CRITICAL - Packet loss = 100% [18:06:39] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Mon Dec 30 18:06:36 UTC 2013 [18:06:49] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Mon Dec 30 18:06:41 UTC 2013 [18:07:19] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 06:06:41 PM UTC [18:07:29] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 06:06:36 PM UTC [18:09:06] <^d> Ok, so what are we gonna do? [18:09:33] i'm here now [18:09:38] I see [18:09:38] sbmit a patch and i'll merge [18:09:43] on it [18:10:19] RECOVERY - Host search1005 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [18:10:59] (03PS1) 10Aaron Schulz: Fix array copy in scap-2 for real this time [operations/puppet] - 10https://gerrit.wikimedia.org/r/104533 [18:11:03] ori: ^ [18:11:29] rsync doesn't whine in test.sh...so that better work [18:12:10] (03CR) 10Ori.livneh: [C: 032 V: 032] Fix array copy in scap-2 for real this time [operations/puppet] - 10https://gerrit.wikimedia.org/r/104533 (owner: 10Aaron Schulz) [18:13:06] * Aaron|home was relying on echo too much to test stuff last week [18:13:40] some of these type errors are invisible when you print stuff [18:14:03] rewrite in haskel [18:14:17] meh [18:14:27] today greg-g is helpful I see [18:14:28] you made like a dozen substantial improvements [18:14:30] breakage is ok [18:14:31] ^d: so even if tin works, all the servers need a puppet run to fix scap-2 [18:14:51] Nemo_bis: hey, type errors would exist :) [18:15:40] Aaron|home and ori: let me know when I should redo it. [18:16:17] manybubbles: k, figuring out a scheme to rsync it so you don't have to wait for the entire app server group to run puppet [18:17:08] ori: cool. I don't mind waiting but I figure I'm a better test then the train [18:17:29] rather, lower cost of failure [18:18:13] yep. sorry 'bout that! [18:20:19] RECOVERY - Puppet freshness on virt1002 is OK: puppet ran at Mon Dec 30 18:20:12 UTC 2013 [18:20:29] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Last successful Puppet run was Mon 30 Dec 2013 06:20:12 PM UTC [18:20:57] <^d> ori: salt? [18:23:51]