[00:25:33] anyone got a minute to check out my borked exported resources ? [00:25:33] New patchset: Asher; "fix package dependency order issue, remove hardy bits" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6360 [00:25:49] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/6360 [00:30:55] New patchset: Asher; "fix package dependency order issue, remove hardy bits" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6360 [00:31:11] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/6360 [00:42:40] New patchset: Asher; "fix package dependency order issue, remove hardy bits" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6360 [00:42:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6360 [00:43:47] finally! [00:51:40] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/6360 [00:51:43] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6360 [01:01:15] starting innobackupex from db57 to db53 for new s2 slave for the one zillionth time [01:01:20] !log starting innobackupex from db57 to db53 for new s2 slave for the one zillionth time [01:01:23] Logged the message, notpeter [08:20:26] !log cherry-picked ae12df0 commit to 1.20wmf2 since there are mobilefrontend commits pending. [08:20:33] Logged the message, Master [08:25:20] New patchset: ArielGlenn; "job status update done in main todo queue so deletes work" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/6369 [08:29:18] Change abandoned: ArielGlenn; "someone (cough) commited this change later on their own" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5887 [08:30:19] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/6369 [08:30:21] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/6369 [08:43:20] !log kaulen: installing various upgrades (apache,mysql,cron,php-wikidiff2,...) [08:43:22] Logged the message, Master [09:20:10] !log upgrading bugzilla to 4.0.6 [09:20:14] Logged the message, Master [09:29:41] New patchset: Mark Bergsma; "Revert "Revert "git::clone define improvements.""" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6372 [09:29:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6372 [09:43:54] New patchset: Mark Bergsma; "git::clone define improvements." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6372 [09:44:12] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/6372 [09:51:59] New patchset: Mark Bergsma; "git::clone define improvements." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6372 [09:52:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6372 [09:52:27] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6372 [09:52:30] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6372 [09:58:18] New patchset: Mark Bergsma; "Remove requirement for removed file resource" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6374 [09:58:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6374 [09:59:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6374 [09:59:20] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6374 [10:09:47] !log Started distribution upgrade of server sockpuppet from Lucid to Precise [10:09:50] Logged the message, Master [10:37:59] New patchset: Mark Bergsma; "Don't install dashboard on sockpuppet, it scales very badly" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6375 [10:38:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6375 [10:38:45] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6375 [10:38:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6375 [10:40:59] !log refreshLinks.php - started it once again in a screen on hume, just for s1. last cron failed with "mwscript command not found"?? well now it is there again and running [10:41:02] Logged the message, Master [11:05:50] mutante, what can I do now to help you with wikistats? [11:06:08] check that list of wikis you published on https://bugzilla.wikimedia.org/show_bug.cgi?id=36268 ? [11:06:29] * Nemo_bis hopes the answers is not "learn git and commit patches" [11:08:17] Nemo_bis: well.. both :) hehe. checking that list would be cool. using git would be awesome, but you don't have to [11:08:53] putting stuff in bugzilla tickets is a first step [11:09:43] Nemo_bis: i would like those URLs replaced with links to api.php or Special:Statistics?action=raw [11:10:01] could script it, but we imported thousands and these are just 56 special cases left [11:10:27] and you never now about special installations having weird URL rewrites and stuff [11:24:01] New patchset: Mark Bergsma; "Add the Dell R300 platform to base::platform" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6378 [11:24:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6378 [11:24:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6378 [11:24:45] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6378 [11:35:32] New patchset: Mark Bergsma; "Dell R300 uses serial speed 57600" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6379 [11:35:50] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6379 [11:35:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6379 [11:36:01] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6379 [11:36:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6379 [11:45:17] i'm going to dist upgrade stafford in a bit [11:45:23] so puppet will be temporarily unavailable [11:46:52] k [11:52:49] !log Started distribution upgrade of server stafford from Lucid to Precise [11:52:52] Logged the message, Master [11:57:59] mutante, those URL are something I could fix with your shell script, right? [12:02:07] Nemo_bis: eh no, the URLs in that bug above are those left the script does not get, because they are nonstandard. so either manually because it is a really low percentage and seems like a one-time job, or enhance script [12:03:03] the script works if you either have an "api.php" or a "Special:Statistics", but not for stuff like "Spezial:" "Speciaal" , language thing [12:03:18] mutante, I meant the script to add an API URL at a time [12:04:00] true [12:04:18] it had a "change URL" feature [12:05:25] these are old additions, you could say we don't accept any non-standard ones like this if somebody wants to add [12:06:47] New patchset: Pyoungmeister; "correecting ian baker's pubkey." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6380 [12:07:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6380 [12:07:38] New review: Pyoungmeister; "safe to be pushed out any time. when stafford is back up." [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6380 [12:07:41] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6380 [12:08:11] which is almost ;) [12:10:26] and how do I know that is safe to be pushed out? ;) [12:12:09] notpeter: where is that RT ticket? [12:15:09] New patchset: Mark Bergsma; "Revert "correecting ian baker's pubkey." No idea how I'm supposed to verify whether this is genuine." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6381 [12:15:18] :-]] [12:15:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6381 [12:15:40] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6381 [12:15:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6381 [12:15:54] mark: we had a long discussion about updating ssh keys with ^demon and Ryan [12:16:03] not sure we wrote a process though :-( [12:17:40] http://rt.wikimedia.org/Ticket/Display.html?id=1329 [12:17:49] http://rt.wikimedia.org/Ticket/Display.html?id=2835 [12:17:54] roan fucked up his key [12:18:02] might be nice to mention those in the commits [12:18:08] I couldn't find them with dumb rt search [12:18:16] feel free to revert my revert [12:18:21] but I'm just making a point :) [12:18:25] did you look at the diff? [12:18:29] yes I did [12:18:32] the difference was == at the end [12:18:35] I noticed [12:19:04] ok I wasn't making a point, I was destroying two = ;-) [12:19:30] I'm going to go drink coffee. [12:25:51] drop table; http://pinterest.com/pin/11610911509612074/ [13:04:17] hashar: how is gallium doing with the query log? [13:04:51] I havent checked [13:04:59] wait [13:05:07] aren't logs written in /var/log/mysql/ ? :D [13:05:13] if so I am 100% sure I can't read them [13:05:44] oh, yeah, you did not specify a different log location for the slow query log right [13:06:17] i commented about that, because i did not see anything in mysql log, at least not right away after the config was changed. looking.. [13:06:52] /var/log/mysql just nothing but error.log and error.log-old [13:07:05] maybe that is logged somewhere else [13:07:15] or there are no slow queries (doubtfull) [13:07:26] yeah, but expected default, there did not seem to be a path in that config [13:08:16] what should i look for besides "slow" [13:10:28] mutante: looks like the default is hostname-slow.log [13:10:35] aka: gallium-slow.log [13:10:44] in the data directory [13:11:03] ah, correct:) [13:11:10] so should be: /var/lib/mysql/gallium-slow.log [13:11:14] you got them [13:11:26] who wants to look at them [13:11:36] me / ^demon I guess [13:12:27] maybe we could write a basic shell scripts wich would allow us to : sudo cat /var/lib/mysql/gallium-slow.log [13:12:47] or have the file written somewhere where we can read from [13:13:08] yeah, didnt you want it in /home/ somewhere [13:15:45] hashar: it does not exist, but /home/wikidev would make sense i guess, with you guys being in that group [13:17:32] hashar: copied gallium-slow.log to your home [13:18:07] danke schoen [13:18:12] de rien [13:18:50] oh Google gave me a link to Ferris Bueller's soundtrack: "Danke Schoen" http://www.youtube.com/watch?v=ru-wvqo1SFY [13:19:06] Ferris Bueller's Day Off is an amazing movie btw [13:19:23] hehe, yea, it is on the list of classics i guess [13:19:40] even a cult one [13:20:40] ^demon: you also got a copy of gallium-slow.log in gallium home [13:21:02] <^demon> Just what I wanted? :) [13:21:18] ^demon: that is a slow query log for Krinkle [13:21:25] he wants to enhance testswarm queries [13:21:45] <^demon> Yeah I knew what it was for, just didn't know that I had asked for a copy :p [13:21:54] oh [13:22:06] I have been bold and figured out you could use one [13:22:18] so I can tell Timo : "no time, ask Chad he got a copy" [13:22:21] :-]]]]]]]]]]]]]]]]]] [13:22:46] * mutante hides ands lets you forward it to krinkle [13:23:49] <^demon> And I'll say to Timo: "Hashar's crazy if he thinks I'm doing his work." ;-) [13:23:55] (is not a user on gallium) [13:24:01] :)))))))))) [13:24:37] New patchset: Mark Bergsma; "Add precise to the default list of pbuilder distributions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6385 [13:24:48] laughed so loud that most people at the coworking place stared at me [13:24:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6385 [13:25:11] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6385 [13:25:14] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6385 [13:28:36] is there a way to configure gerrit to only show some projects without having to modify your search for it all the time? :) [13:28:43] I don't want to see all the mediawiki crap ;-) [13:33:27] <^demon> mark: Upstream feature request to "save searches" [13:34:08] awesome [13:36:15] or we could disallow you access to mediawiki projects :-] [13:37:50] <^demon> That could work too ;-) [13:38:24] that's a good idea [13:38:27] I'll remove my access [13:39:00] <^demon> Well we'd have to make a new group, put mark in it and then set DENY -> READ on mediawiki/* [13:41:11] group could be named something like `pureops` `top-of-food-chain` [13:42:02] <^demon> `people-bothered-by-clutter` [13:42:18] people-who-hate-web-scripts [13:42:28] ;) [13:43:54] heyaaaaa [13:44:08] mark, if you got a sec, could you tell me about the different varnish::logging stuff in puppet? [13:44:13] there seem to be two ways to set things up right now [13:44:17] but one isn't really used [13:44:18] yeah [13:44:34] one is dirty, one is cleaner but wasn't working [13:44:37] then notpeter did some work on it [13:44:41] and right now i'm not sure which one is used [13:45:01] varnish::logging is used [13:45:04] i think it is the old one [13:45:06] using the init.d script [13:45:16] varnishncsa is the new one, using upstart [13:45:23] just modify the format string in both [13:45:25] and you're safe [13:45:48] well, i'm having trouble modifying it in the init script, due to really annoying start-stop-daemon argument quoting [13:45:48] upstart was a bit problematic in lucid [13:45:55] but i'm just now working on getting varnish on precise [13:45:58] so it migth work better then [13:46:09] i feel like i'm doing something dumb, but andrewbogartt and I looked at it for a while then, and had so much trouble [13:46:16] I can do it in an in elegant way [13:46:24] and make a special variable in the init script [13:46:28] that I can set in /etc/defaults [13:46:43] hm, ok [13:46:52] if the upstart stuff doesn't really work [13:46:55] <^demon> mark: The other good way is to avoid using All -> Changes and use My -> Changes and My -> Watched Changes more. If you add operations/puppet to your watchlist in your settings, they'll show up in Watched Changes. [13:47:06] alright [13:47:20] i guess i'll do my inelegant way, with the hope that varnish puppet will be cleaned up later [13:47:26] i was gonna ask if I should work on making it clean now [13:47:32] buuuut, if upstart doesn't work right..meh? [13:47:51] you can try ;) [13:47:53] or you can wait [13:49:09] well, i'd try if upstart worked right in lucid [13:49:11] dont' want to fight that [13:49:31] yup but at least those varnish boxes won't be on lucid for much longer [14:03:56] Could someone put http://noc.wikimedia.org/~reedy/upload-1.19.0.tar on dataset2:/data/xmldatadumps/public/mediawiki and then extract it please? [14:05:48] <^demon> That really needs a new home. [14:22:13] Indeeeed [14:23:57] sometimes I still don't get git at all. [14:26:48] Reedy: check that it looks ok please [14:29:31] apergos: looks good. Thankyou [14:29:38] sure [14:35:19] we should get jenkins to build deb packages for us [14:35:51] <^demon> That's doable. [14:36:04] <^demon> Jenkins can do pretty much anything if you can script the process. [14:36:40] the script is called git-buildpackage ;) [14:37:56] <^demon> Yeah, all we'd need is a build.xml for ant to know what to do. [14:40:10] ant is not required [14:40:21] we could use rake / a shell script / a c program :-] [14:40:23] and some standard on what distro to build it for based on branch name [14:40:39] and/or releases vs snapshot builds [14:40:49] the good thing with ant is that Jenkins as additionals and nice supports for it [14:40:51] although I guess the latter would ideally stay outside of that [14:42:09] <^demon> https://wiki.jenkins-ci.org/display/JENKINS/Plugins has so many toys we can look at too. [14:42:21] <^demon> Might be some people who've done the heavy lifting already and plugin-ized it. [14:43:46] !log Built varnish for precise as 3.0.2-2wm5 and imported it into APT repository precise-wikimedia [14:43:50] Logged the message, Master [14:43:52] <^demon> There's also a couple of publish/deploy plugins that are kind of cool. [14:44:22] and the winner might be : https://github.com/mika/jenkins-debian-glue#readme [14:45:01] * mark adds it to his list as a berlin hackathon topic [14:45:14] <^demon> hashar: Ooh, that looks cool. [14:45:15] bug report it ? :-D [14:45:30] that would mean I'd need to figure out how bugzilla works [14:45:41] Wait, bugzilla works? [14:45:48] no clue [14:46:03] ohhhhhhh [14:46:57] since the author is offering consulting services, we could just have him set it up for us ;-]] [14:47:59] I could just have you set it up for us. [14:48:06] :-))) [14:48:09] notpeter: Good morning. So what's the RT ticket situation exactly? [14:48:12] probably going to do that [14:50:51] mark: bug request is https://bugzilla.wikimedia.org/show_bug.cgi?id=36443 [14:51:15] CCed you [14:51:49] my internet is sllowwwww [14:51:52] still loading [14:52:21] ty :) [14:56:01] New patchset: Mark Bergsma; "varnish (3.0.2-2wm5) precise-wikimedia; urgency=low" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/6391 [14:56:26] New patchset: Ottomata; "RT 2255 - adding Content-Type response header to varnish log lines." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6392 [14:56:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6392 [14:56:54] mark, could you check that one for me? I don't like how I did it but I think it is good enough for now [14:57:37] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/6391 [14:57:39] i gotta run for 15 mins, be back in just a bit, leave comments in gerrit [14:57:39] Change merged: Mark Bergsma; [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/6391 [14:58:33] OH [14:58:34] mark! [14:58:38] oh nm [14:58:44] that is varnish, not varnishncsa [14:58:53] if you are rebuilding varnishncsa for precise [14:58:54] can we make a change to the default log line? [14:59:05] ack, brb [14:59:48] New review: Mark Bergsma; "Has this been tested?" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/6392 [15:00:00] varnishncsa is part of varnish [15:00:26] we could, but this should be configurable anyway [15:00:36] we don't want to roll a new varnish build every time a new log format is needed [15:01:06] New patchset: Catrope; "Resubmit "correecting ian baker's pubkey."" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6393 [15:01:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6393 [15:01:27] robh: can you please take Bellin out of rotation for hardware test [15:01:33] !rt 2872 [15:01:33] https://rt.wikimedia.org/Ticket/Display.html?id=2872 [15:02:29] cmjohnson1: ok, so notpeter put in this ticket [15:02:29] robh: also mw64 [15:02:36] !rt 1890 [15:02:36] https://rt.wikimedia.org/Ticket/Display.html?id=1890 [15:02:41] so when its something of his, feel free to ping him if he is aobut [15:02:46] ok [15:02:56] mostly cuz he would know if its still doing work or is out of cluster [15:03:08] but i am checking, cuz he may not be responsive at the moment [15:03:27] it looks to be offline [15:03:56] !log bellin crashed, unresponsive to ssh or serial console [15:03:58] Logged the message, RobH [15:04:06] cmjohnson1: so bellin is crashed, offline, you can do what you need to with it [15:04:56] !log shutting down mw64 for hw test per rt 1890 [15:04:58] Logged the message, RobH [15:05:05] robh: thx [15:05:10] cmjohnson1: ok, all yours [15:07:17] cmjohnson1: thank you! [15:08:30] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6393 [15:08:33] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6393 [15:09:20] cmjohnson1: after a lot of head-scratching, I realized that it couldn't be done over console redirection [15:09:30] cmjohnson1: and yeah, ask me anything you like! [15:12:42] RobH: are you in the dc? [15:13:19] nope, but plan to be later this week [15:13:20] hmm actually it doesn't matter nm, these hosts aren't there [15:13:23] need something tested? [15:13:34] mark: could you potentially review https://gerrit.wikimedia.org/r/#change,5813 ? It is to let the labs people send Apache syslog to a specific non-production server [15:13:40] no, we probably need someone to look at a few hosts and see whats up with them [15:14:07] notpeter: i see where you tried [15:15:07] heh [15:16:37] hmm maybe someone got to them already [15:16:38] nice [15:21:23] mark: thanks! [15:24:07] New review: Ottomata; "Mark, yes, on my local VM. I tried so many different ways of making this more elegant, and this is ..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/6392 [15:24:51] is bits.wikimedia.org have problems again? [15:25:23] Jeff_Green: ^^ [15:25:54] It is not loading on: [15:25:55] https://www.mediawiki.org/wiki/Special:Code/Wikimedia/1662 [15:26:51] jpostlethwaite: this is the first I've heard of anything [15:27:04] i loaded that once and it came up without the bits, then with bits the second time [15:27:22] It just started for me [15:27:29] ok [15:27:32] bits is loading [15:27:34] for me [15:27:48] thanks :) [15:27:55] huh [15:28:06] lemme know if you notice it again [15:28:20] notpeter or apergos, who wants to help me figure out why I can't write to the pagecounts-ez rsync module on dataset2 from stat1? [15:28:45] did youget someone (e.g. me) to create a writeable directory over there for testing? [15:28:50] I bet you forgot to do this [15:29:05] needs to be writeable by the user specified in the module [15:29:24] hm, ok [15:29:37] can you check then? the module is running as backup i think [15:30:04] yes it is [15:30:09] /data/xmldatadumps/public/other/pagecounts-ez [15:30:11] is the directory [15:30:18] there is now a directory "testing" in there that you can write to for... testing, duh [15:30:20] what are perms/ownership on that? [15:30:24] ok [15:30:28] it's all owned by ezachte [15:30:35] because he writes in there directly right now [15:30:37] Reports of bits issues in -tech [15:31:01] when you're ready to move, as I mentipned earlier, we'll switch perms on the entire dir recursively to the right user [15:31:07] !log no nagios bot, kicking nagios on spence [15:31:09] Logged the message, notpeter [15:31:17] cool!, that works, hm ok [15:31:24] happy script-writing [15:31:43] ezachte is going to be rsyncing from stat1…if we make rsync module gid=wikidev and chmod -R g+w on that dir [15:31:48] would that be good enough? [15:32:03] maybe chgrp -R wikidev as well? [15:32:14] Reedy: bits was out for a few minutes for me and came back up [15:32:24] because uh, we are ready to move :) [15:32:27] I'd rather just make the owner/group be backup on the dir contents [15:32:31] i could not load code review [15:32:34] just this and a DNS change and stat1 will be finished [15:32:40] that's fine with me [15:32:43] so if he wants to do that now I'll do the chmod right this second [15:32:47] it appears to be working now [15:32:52] aye beacuse he might be using nfs from bayes [15:32:58] hm, can we backup:wikidev it? [15:33:07] g+w, just in case? [15:33:31] ottomata: do you want to upgrade stat1 to precise? [15:33:34] I thought we want to not do nfs any more [15:33:37] um, sure? [15:33:41] right we don't [15:33:45] while you're still working on it [15:33:46] but since we are not 100% using stat1 yet [15:33:50] and today is report card metrics meeting day [15:33:52] saves everyone a later upgrade [15:33:58] he might be using bayes and writing there [15:33:59] i doubt it though [15:34:05] but if we make it g+w :wikidev [15:34:15] then at least he can still write on bayes (which will be deprecated once stat1 is finished) [15:34:33] when can we decommission bayes you think? [15:35:12] I don't want to leave it that way [15:35:28] when do we expect him to be using stat1 exclusively? [15:37:29] let's wait with upgrading to precise until friday [15:37:38] for stat1? [15:37:40] yes [15:37:42] I thought it wasn't used yet? [15:37:47] it is [15:38:08] as of last week, erik z and andre engels are running their scripts on stat1 [15:38:10] I'm hoping for a reinstall instead of an upgrade btw [15:38:16] because that's also a good test for the puppet manifests [15:38:29] fine either way [15:38:37] but it can wait until friday of course [15:38:41] ottomata? [15:38:42] no rush [15:38:53] just saying, now would be a good time to upgrade while it's all fresh on everyone's mind [15:38:57] instead of a year from now or so ;) [15:39:25] grrr [15:41:27] apergos: (about using stat1 exclusively) and I assume you are referring to Erik Z: as soon as we know 100% that stat1 has all the functionalities, libs, data etc that bayes has and that erik can generate everything without problems [15:41:40] is there any eta? [15:42:04] in other words, should I ask again in e.g. a week? 2 weeks? [15:42:05] i would think within 2 weeks, [15:42:08] ok great [15:42:09] or less [15:42:10] * apergos makes anote [15:42:27] just keep poking me and erik z, i will also keep reminding him that we really want to move away from abyes [15:42:59] yes indeed [15:43:45] awesome [15:44:00] so after friday we can reinstall stat1 [15:44:09] then it can start to be used [15:44:18] and then after a week or two, we can decommission bayes [15:44:20] right? [15:44:25] if all goes well of course [15:44:27] yes [15:44:31] cool [15:44:32] (if all goes well) [15:44:52] maybe we should email this to both erik z, andre engels, me, andrew, and apergos [15:44:57] so everybody is in the loop [15:45:06] yes [15:45:11] fine by me [15:45:20] and since that's your excellent idea, i'll let you have it ;-) [15:45:39] love it [15:46:55] mark: I have made the $syslog_server determined in realm.pp and base::remote-syslog a class with param https://gerrit.wikimedia.org/r/#change,5813 [15:48:25] there are multiple errors in there [15:48:30] first, you can't reassign variables in puppet [15:48:40] second, you're mixing a class parameter with a global variable [15:48:50] i'm not sure what's that supposed to do :) [15:49:38] ah I did not know we could not reassign variables :( [15:49:55] about mixing, do you mean $syslog_server and $::syslog_server ? [15:53:12] Reports of bits 503s in #wikimedia-tech [15:57:24] notpeter: can we change the warning for inactive filters from 6 to 18 or 24 hours? [15:57:33] right now we are getting way too many false warnings [15:58:31] drdee_: sure [15:58:39] thx [15:58:59] ag, internet at this cafe is pooopy [15:59:09] missed everything since :34 of this hour [15:59:38] so apergos, if you responded to me I missed it [16:00:15] it's because your response to me never made it here [16:00:22] what you missed was: [16:00:27] mark: updated with at patchset3 https://gerrit.wikimedia.org/r/#change,5813 [16:00:30] we expect him to be off of bayes in a couple weeks [16:00:41] mark: though I need to run out of coworking space right now :/ Will follow up later tonight [16:00:44] an email will go around summarizing stuff like the install timeframe etc [16:01:05] ? [16:01:07] that's not puppet syntax hashar ;) [16:01:10] i said: [16:01:10] he might be using bayes and writing there [16:01:10] i doubt it though [16:01:10]