[00:12:39] RECOVERY - MySQL Replication Heartbeat on db44 is OK: OK replication delay 0 seconds [00:13:06] RECOVERY - MySQL Slave Delay on db44 is OK: OK replication delay 1 seconds [03:03:19] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [03:03:19] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [03:19:22] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [03:19:22] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [04:42:57] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [04:44:18] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [05:32:46] is it just me or is http://wikitech.wikimedia.org/ down? [05:33:44] PROBLEM - udp2log log age on emery is CRITICAL: CRITICAL: log files /var/log/squid/e3_necromancy_idle1year.log, /var/log/squid/e3_necromancy_idle3month.log, have not been written to in 6 hours [05:39:40] pgehres: works for me... [05:40:19] and so it does for me now again [05:40:36] paravoid: thanks for checking [05:51:00] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [05:53:15] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [06:02:15] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [06:04:03] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [06:38:39] RECOVERY - udp2log log age on emery is OK: OK: all log files active [06:47:29] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [07:00:32] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [07:56:55] hello there [08:58:52] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [12:17:14] PROBLEM - Host ssl1002 is DOWN: PING CRITICAL - Packet loss = 100% [12:19:42] hashar: hi. gallium has an issue with Misc::Contint::Test::Testswarm/File[/var/lib/testswarm/mediawiki-trunk [12:19:53] duplicate file [12:20:05] PROBLEM - LVS HTTPS on bits-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:06] PROBLEM - LVS HTTPS on wikimedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:16] oh [12:20:52] mark: you about? [12:21:01] yes [12:21:08] yea, whats up with ssl1002 [12:21:09] is that you changing stuff? [12:21:11] hi all [12:21:11] no [12:21:14] ok [12:21:17] PROBLEM - LVS HTTPS on wikipedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:23] hmm [12:22:10] pybal is not depooling ssl1002 [12:22:43] mutante: oops [12:22:54] !log Manually depooled down ssl1002 in pybal [12:22:56] Logged the message, Master [12:23:39] ok, I'm going to get back to waking up... [12:23:59] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 183 seconds [12:24:08] RECOVERY - LVS HTTPS on wikipedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 79291 bytes in 0.279 seconds [12:24:08] RECOVERY - LVS HTTPS on bits-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 3932 bytes in 0.112 seconds [12:24:08] RECOVERY - LVS HTTPS on wikimedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 79291 bytes in 0.246 seconds [12:24:48] can someone look at ssl1002 at some point? [12:25:02] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 205 seconds [12:25:43] connects to mgmt [12:27:14] !log powercycling frozen ssl1002 [12:27:16] Logged the message, Master [12:28:25] mutante: what is the root cause for the duplicate file ( Misc::Contint::Test::Testswarm/File[/var/lib/testswarm/mediawiki-trunk ) [12:28:29] RECOVERY - Host ssl1002 is UP: PING OK - Packet loss = 0%, RTA = 26.56 ms [12:28:34] mutante: greping syslog does not give me any message :-( [12:30:28] getting a snack for lunch. will be back soo [12:30:29] n [12:32:42] mark: it's back, nginx running, looks normal, nothing obvious. syslog just stops in the middle of normal stuff at 12:15. i guess you could repool it [12:33:06] hashar: /var/log/daemon.log [12:33:16] !log repooled ssl1002 [12:33:18] Logged the message, Master [12:33:23] what worries me is that pybal didn't depool it [12:34:21] mutante: I do not have access to that [12:34:30] (should have applied for a junior ops position :D ) [12:34:51] then nnark would have scratched his head even more! [12:34:57] hashar: in labs we just made "common logs" readable for non-roots [12:34:59] all the cool stuff happens in ops [12:35:27] mutante: anyway, the whole /var/lib/testswarm/mediawiki-trunk should no more be there. I have marked it as ensure=>absent [12:38:11] hashar: are you on gallium? [12:38:18] yup [12:39:44] PROBLEM - udp2log log age on emery is CRITICAL: CRITICAL: log files /var/log/squid/e3_necromancy_idle1year.log, /var/log/squid/e3_necromancy_idle3month.log, have not been written to in 6 hours [12:40:47] mutante: starving for food. give me 10 minutes ;) [12:40:47] hashar: cat /home/hashar/daemon.log [12:40:59] oh reading [12:41:01] sure, i want coffee too [12:41:02] * hashar dies [12:41:16] k so I get a snack and you get coffee [12:41:20] sounds like a great plan [12:57:25] hashar: grep Filebucketed [12:58:15] there are all those files in mediawiki-trunk/checkouts [12:58:49] mutante: back [13:01:25] grep duplicate and you see the md5 sums of the duplicates .. its a long file :p [13:01:50] the first line I got is: Recursively backing up to filebucket [13:02:03] looks like puppet do a recursive copy of the content of /var/lib/testswarm to some place [13:02:10] yes [13:02:21] then takes md5 fingerprints to check if the file already exist in the bucket [13:02:31] there is "recurse" somewhere in the contint class then [13:03:35] maybe [13:03:52] anyway the whole /var/lib/testswarm/mediawiki-trunk/ should be gone by now [13:04:02] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [13:04:02] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [13:04:24] mutante: I have marked the path as ensure=>absent [13:05:02] yea, i saw that.. and found it normal [13:05:05] hmm [13:05:15] running puppet again [13:05:48] relevant change is https://gerrit.wikimedia.org/r/#change,4364 [13:05:53] commit 9ff43e547f995fe1fa12896be5b8e545a5960fb4 [13:06:10] well, it is running all the time.. because it sits there going through all those duplicates [13:06:16] hey hashar , today I'm working sane european hours for a change :) [13:06:25] merged in git yesterday, maybe you forgot to pull it in puppet master ? [13:06:57] Krinkle: is that every Thursday or just this week ? :) [13:08:06] Every thursday [13:08:10] hashar: you know what, this is not actually a broken puppet , it is not a "duplicate file/resource" thing that breaks a puppet run [13:08:24] 12-19 instead of 19-02 [13:08:42] hashar: it is just running for a looooong time, and FileBucket telling us it finds duplicate files [13:09:02] mutante: we should just disable that [13:09:33] mutante: some /var/lib/testswarm paths are holding full mediawiki copies which are going to be a nightmare for any recursive job [13:10:08] I am going to delete some [13:10:20] meanwhile can you have a look at https://gerrit.wikimedia.org/r/4743 ? :-] [13:10:23] it is for gallium too [13:10:30] now that its running i am not sure if we should just let it finish [13:10:45] I can delete the file and then have puppet to rerun? [13:10:50] might be faster I don't know [13:11:32] * hashar deleting [13:11:40] yeah, but whats better? killing it or just letting it do the job [13:12:13] as long as its not keeping you from using the services... [13:14:41] load average: 8.34, 4.60, 2.91 [13:14:42] hehe [13:18:00] it's recursive delete via puppet, yea. reading through some closed bugs .. but http://projects.puppetlabs.com/issues/3835 http://projects.puppetlabs.com/issues/3180 ..but yeah, it does NOT say there is a problem setting anything to absent, it just tells us it finds a lot of files in /.svn/ with duplicate md5 sums.. and takes a long time to do that [13:18:24] cause it has to md5sum all the millions of files there [13:18:42] it is still doing its job though [13:18:47] then most probably create a a file bucket with a huuuuuuge index [13:19:13] so I guess you should just ^C puppet [13:19:27] I have launched the rm commands to delete the files [13:20:07] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [13:20:07] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [13:21:11] hmm, ok.. [13:21:19] PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 9.09349632813 (gt 8.0) [13:21:55] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [13:22:10] Change abandoned: Pyoungmeister; "file has changed significantly. going to redo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2083 [13:22:37] doh a bit more than 60 secs of real time to delete a directory [13:22:43] and there is a thousand of them [13:23:03] puppet agent would not react on regular stop anymore anyways [13:23:07] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [13:23:13] !log killed puppets on gallium [13:23:16] Logged the message, Master [13:26:56] mutante: can you possibly review https://gerrit.wikimedia.org/r/#change,4743 ? [13:27:13] looked at rm working hard [13:27:22] looking [13:34:45] wait, what? [13:34:47] what are you guys doing? [13:34:52] deleting tons of files with puppet!? [13:37:41] I got million of files in /var/lib/testswarm/mediawiki-trunk/ [13:37:50] that holds various checkouts of mediawiki source files [13:37:51] we stopped puppet [13:38:02] recently had puppet to ensure=>absent that directory [13:40:34] I ber, atleast a million [13:40:48] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/4743 [13:40:50] 1100 checkouts roughl [13:40:51] complete svn checkouts of every rev in trunk/phase3 for several months [13:41:13] and with all the .svn metadata [13:41:13] hashar: I see why you want to leave out .git now :) [13:41:21] :-]] [13:41:28] although leaving that out does mean mw won't be able to get gitinfo [13:41:35] I currently do a recursive copy excluding .git [13:41:35] but whatever [13:41:40] but will have to use git clone instead [13:41:44] nah [13:41:46] it does hardlinks [13:42:04] static snapshot of files related to mediawiki should be fine [13:42:31] hashar: btw, what's the status? anything I can do to help? [13:42:45] otherwise I'll pick up the next section to implement for testswarm [13:42:58] "rm" on gallium still going ... [13:43:21] oh.. [13:43:22] xD [13:43:24] Krinkle: maybe the addJob() script [13:43:25] the testswarm user does it, yea [13:44:23] mutante: I am confused by your comment on 4743 about setting the db directory to 2775 [13:44:59] I need members of the www-data group to be able to write to the mediawiki-git/db dir [13:45:05] and old dirs contained there [13:45:19] the sticky bit is merely to make sure any subdirs are in the www-data group [13:45:22] puppet adds the +x on directories [13:45:28] so you just need a 6 [13:46:01] well given that declaration is only on a directory and not recursive, I think it is better to directly fill the real rights [13:47:07] but I can move it to 2664 , that will avoid a mistake if we ever make that declaration recursive :-D [13:49:39] ok, yeah. i dont know about the part that sqlite databases need to be writable by apache [13:50:11] they do need to be writable :D [13:50:24] cause we are going to have MediaWiki (run under apache) to insert data in the sqlite databases [13:50:37] apache running as apache:www-data , it would be able to do so [13:50:44] Jenkins will take care of g+w [13:51:43] ok. makes sense [14:03:33] that is still rm -f ing [14:14:38] New review: Hashar; "(no comment)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/4743 [14:16:36] hashar: so 2664 , i checked for the SGID bit in puppet because in the type reference it does not mention it. it works as expected and i got to try with "puppet apply". thats a nice way to test [14:16:53] echo 'file { "/tmp/foo_puppet_test_123": ensure => directory, mode => 2644; }' | puppet apply [14:17:37] directory created in there belongs to group [14:26:13] http://docs.puppetlabs.com/references/2.7.0/type.html#file (at "mode" still has that 3 digit example and about puppet adding +1 but the "quite limited" description and does not mention SGID etc. then on latest same section, it now says should use four-digit notation and mentions setuid/setgid/sticky .. [14:26:40] http://docs.puppetlabs.com/references/latest/type.html#file [14:29:21] mutante: whhaaa I never heard of puppet apply [14:29:28] sounds like a great way to test stuff :D [14:30:01] about the mode, it is indeed supposed to be 4 digits [14:30:16] puppet-lint complains about mode using 3 digits [14:30:56] i got that from puppet channel . yes, nice to test [14:34:47] so /bin/ls -r1 |xargs -P16 -n1 rm -fR [14:34:50] still running [14:35:12] New patchset: Demon; "Adding some extra debugging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4781 [14:35:25] ext3 is sooo slow at deleting files :-( [14:35:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4781 [14:35:39] <^demon> Someone mind approving ^ so I can do some debugging on manganese? [14:36:03] commit message is lame, does not reference a bug number, also messages.inc [14:36:17] <^demon> messages.inc? [14:36:43] sorry copy pasted my standard reply [14:36:57] New review: Hashar; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/4781 [14:36:59] looks good to me [14:37:13] though you probably would want to log any parameters passed to the hook [14:37:36] New review: Asher; "(no comment)" [operations/debs/varnish] (patches/udplog); V: 0 C: 1; - https://gerrit.wikimedia.org/r/4729 [14:39:10] <^demon> I'm logging options.project, options.branch and options.uploader. I don't need the rest. [14:40:17] k :) [14:40:26] so now you need someone to merge the apply :-D [14:40:39] <^demon> Yeah, I need an opsen :\ [14:42:02] New review: Dzahn; "2 does what it should. SGID tested with puppet apply." [operations/puppet] (production); V: 1 C: 0; - https://gerrit.wikimedia.org/r/4743 [14:42:34] ^demon: at this time there is pgehres but is probably still asleep today [14:43:17] Jeff_Green maybe you are available to have a change merged? :-) https://gerrit.wikimedia.org/r/4781 [14:43:55] New patchset: Pyoungmeister; "udp2log cleanup, new instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4782 [14:44:05] hashar: looking [14:44:08] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4782 [14:44:51] <^demon> Jeff_Green: Thanks. I'm trying to debug something and this is logging stuff to make that easier :) [14:44:56] sure [14:45:21] mutante: want me to do 2664 ? [14:45:33] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/4781 [14:45:45] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/4781 [14:45:48] hashar: yes please [14:46:02] "requires code review" [14:46:12] I haven't run into that yet--what does it mean? [14:46:20] <^demon> Could you not +2 it? [14:46:50] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4781 [14:46:50] mutante: note that I have used 2775 for /var/lib/jenkins a few step above :-D [14:46:53] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4781 [14:47:19] ah, I thought my +1 would combine with hashar's to make for +2 action [14:47:22] <^demon> Nope. [14:47:36] +1 * n < +2 [14:47:52] that's some crazy math [14:47:54] we need that +2 message to be shown as a check mark or something [14:47:56] hashar: arr..hmm.. what should i say, we need our own style guide? :p [14:48:07] <^demon> hashar: +2 is show as a check mark after you click it :) [14:48:16] mutante: I propose to keep the 2775 to have consistent modes across the file [14:48:30] mutante: then we can submit another change that change all 2775 to 2664 :-D [14:48:43] so if 5 people all clicked +1 you would have a score of 5? naaahhhh [14:48:52] apergos: nooooooo [14:48:59] apergos: you still are at the +1 state [14:49:01] it seems to me the +1's are token at that point [14:49:03] I know [14:49:03] New patchset: Mark Bergsma; "No longer apply the previous varnishncsa patch" [operations/debs/varnish] (patches/udplog) - https://gerrit.wikimedia.org/r/4783 [14:49:04] New patchset: Mark Bergsma; "Fix compile errors" [operations/debs/varnish] (patches/udplog) - https://gerrit.wikimedia.org/r/4784 [14:49:05] New patchset: Mark Bergsma; "glibc ignores setvbuf's size argument in some cases" [operations/debs/varnish] (patches/udplog) - https://gerrit.wikimedia.org/r/4785 [14:49:09] well they aren't [14:49:15] apergos: i know you know :-D [14:49:19] hashar: ok, convinced, since technically it doesnt make a difference, you're right about keeping it consistent across file [14:49:28] the idea if there are several people reviewing is that yo can see who lkes a change and who doesn't to what degree, [14:49:29] apergos: is a +1 required in some contexts for a subsequent +2 to go through? [14:49:32] <^demon> Nope. [14:49:34] mutante: :-]] [14:49:35] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (patches/udplog); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4729 [14:49:37] Change merged: Mark Bergsma; [operations/debs/varnish] (patches/udplog) - https://gerrit.wikimedia.org/r/4729 [14:49:48] Jeff_Green: not that I have ever seen [14:49:49] <^demon> +1's are designed so people can review code without approving it. [14:49:58] so it's just a memo [14:50:03] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (patches/udplog); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4783 [14:50:10] Jeff_Green: in MediaWiki we have volunteer to +1, so when you have some familiar names that +1 stuff you can sometime assume the change is fine [14:50:15] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (patches/udplog); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4783 [14:50:17] Change merged: Mark Bergsma; [operations/debs/varnish] (patches/udplog) - https://gerrit.wikimedia.org/r/4783 [14:50:20] well just a memo would be with no score [14:50:33] the score actually has a (somewhat subjective) meaning [14:50:44] <^demon> "Looks good to me, but someone else must approve?" [14:50:46] we could replace the +1 with a thumbs-up icon, and the +2 with a check icon [14:50:47] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (patches/udplog); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4784 [14:50:50] no, wait, if 5 people clicked +1 you would be higher on the list of patchsets than the one that just has one +1 [14:50:56] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (patches/udplog); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4784 [14:50:58] Change merged: Mark Bergsma; [operations/debs/varnish] (patches/udplog) - https://gerrit.wikimedia.org/r/4784 [14:51:03] ahah [14:51:08] if the 5 peple who clicked have any knowledge about the code [14:51:08] mutante: oh interesting, I didn't realize there was a ranking thing in play [14:51:14] if 5 random people clicked... [14:51:16] indeed, +1 can be seen as a bumping status [14:51:20] <^demon> mutante: I thought it was just based off latest review, not number of reviews. [14:51:22] though -1 will keep it on top too [14:51:25] the logging change is deployed [14:51:34] ^demon: deployed! [14:51:34] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (patches/udplog); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4785 [14:51:37] Change merged: Mark Bergsma; [operations/debs/varnish] (patches/udplog) - https://gerrit.wikimedia.org/r/4785 [14:51:39] thanks jeff [14:51:42] <^demon> Jeff_Green: Thank you. [14:51:48] np [14:51:53] maybe i was just expecting that to be score, and got it wrong, because until now not many sets got more than one +1 [14:52:10] but the one that has the latest comment is back up in the list for sure [14:53:01] <^demon> I think it's the comment that bumps it :) [14:53:07] how can one literal quotes/double quotes in a puppet manifest? specifically, in an array/hash? [14:53:26] shell out to perl and use qw// ? [14:54:39] heh [14:54:45] * mark stabs hashar [14:55:05] notpeter: what do you mean? just by escaping? [14:55:14] yeah [14:55:18] * hashar fall on floor [14:57:32] New review: Dzahn; "ok, convinced to keep it consistent across the file since the 755 vs. 644 discussion is just style w..." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4743 [14:57:35] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4743 [14:58:10] mutante: we still have to wait for the files deletion to end on gallium :/ [14:58:29] New patchset: Pyoungmeister; "udp2log cleanup, new instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4782 [14:58:33] <^demon> Jeff_Green: Could you also force a puppet run on manganese? Puppet ran just minutes before the change :p [14:58:34] yes, but puppet is stopped, so we could do this anyways [14:58:36] hashar: [14:58:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4782 [14:59:05] then it will attempt to file bucket all the files in /var/lib/jenkins/mediawiki-trunk [14:59:17] oohh let me rename it [14:59:49] ^demon: ya, done [14:59:53] mutante: I have renamed the directory, that should hides it from puppet view [15:00:33] <^demon> Jeff_Green: Ok, changes live now, thanks. [15:00:55] wtb puppet broadcast changes feature [15:02:02] New patchset: Pyoungmeister; "udp2log cleanup, new instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4782 [15:02:18] Project does not begin with mediawiki/ [15:02:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4782 [15:02:46] mark: would you be willing to look at ^ ? [15:02:58] or binasher [15:02:59] sure [15:03:13] ct asked me to spin up oxygen the rest of the way [15:03:23] I thought it was a nice excuse to finally finish my udp2log conf cleanup [15:03:42] hashar: running puppet [15:03:45] and watching it [15:04:12] File[/var/lib/testswarm/mediawiki-git/db/]/group: group changed 'jenkins' to 'www-data' [15:04:13] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Thu Apr 12 15:03:37 UTC 2012 [15:05:06] hashar: it is also changing mobile nightly builds from "jenkins" to "wikidev". . and it finishes fine [15:05:12] ok [15:05:16] the NRPE stuff can be done in a nicer way now [15:05:38] cool! [15:05:47] with nrpe::monitor_service [15:05:51] ah, yes, right! [15:06:00] that will setup the nrpe.d/ file for you etc [15:06:21] you may want to rename /etc/udp2log/squid to something more appropriate while you're at it [15:06:22] yep [15:06:24] (cdn or so) [15:06:28] i'll take a close look soon [15:06:36] mark or anyone--I'm failing to find the knobs to control file/directory attributes when packaging a deb, is there hidden magic somewhere or is it usually done with a makefile? [15:06:37] sure, cdn is fine [15:06:39] you've already seen it mutante [15:06:54] ah,ok, sounded like you made another change just now [15:07:01] Jeff_Green: it just takes them as they are installed by the debian/rules file [15:07:07] usually with "install" commands in there [15:07:18] mutante: http://integration.mediawiki.org/clone/mw/master/test.txt [15:07:22] mutante: thanks! :-] [15:07:30] hashar: :) [15:07:42] notpeter: just wondering, is there any good reason to list all filters in a puppet array instead of simply in the config file? [15:07:58] to not have a config per host? [15:08:07] and let puppet populate it from the array [15:08:12] New review: Hashar; "I have tested the deployment and it works well. Thanks!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4743 [15:08:20] that's nice if puppet can do other cool stuff with it [15:08:31] if it doesn't, I would say, just keep it simple [15:08:35] ok [15:08:38] fair enough [15:08:41] not that I object to this [15:08:43] still simple enough [15:08:47] just wondering what you were planning to do with it [15:09:03] you could for example automatically setup monitoring for each filter or something like that [15:09:04] mark: huh, ok I'll look more closely there [15:09:42] yeah. the monitoring I have parses that conf file... [15:09:52] so I could probably reuse and make it a little less janky [15:10:09] also [15:10:16] can we get rid of that aft thing? ;) [15:10:23] !log gallium - after files have been deleted/moved, puppet back to normal operation (and new clone directory in Apache) [15:10:23] it looks more like a separate instance of udp2log [15:10:25] Logged the message, Master [15:10:29] it is [15:10:29] right? [15:10:31] yeah [15:10:37] then it would be cleanest if you made generic multiple instances support [15:10:41] much like we do with varnish [15:10:46] ah, yes [15:10:49] makes sense [15:10:50] then we can easily add more instances [15:10:56] i'm not sure how necessary this is [15:11:08] so perhaps you shouldn't spend a lot of time on it [15:11:11] but it would be cleanest [15:11:25] shouldn't be too hard [15:11:35] perhaps multithreaded udp2log is around the corner, and we may not need multiple instances at all [15:11:36] I don't know [15:11:53] (afaik it's only done for performance reasons atm) [15:14:06] ok, cool. I shall make these changes [15:14:23] as long as I'm cleaning tings up, might as well put in the extra hour to actually make it nice [15:14:32] \o/ [15:18:18] hashar: one small thing is left. "File[/etc/testswarm]: Not removing directory; use 'force' to override". [15:18:48] hashar: backend uses mediawiki-git fronted says ./mw/master which is correct? [15:19:10] mutante: indeed, we have to keep /etc/testswarm it is used by the debian package [15:19:23] also, should wgScriptPath include {changePath} >? [15:19:24] ok, i figured [15:19:45] seeing the config.Debian.ini [15:20:07] I need to revert that part [15:20:23] Krinkle: mediawiki-git is a filesystem path [15:20:24] probably should since it is asynchronous and even if not, it has to be cache resident, every changeset a different url [15:20:41] I know, but this thing isn't limited to mediawiki/core:master right ? [15:20:53] Krinkle: /clone/mw/master is an Apache alias used by clients (aka an URL like http://integration.mw.org/clone/mw/master/1234/1/index.php?title=foobar [15:20:56] so why /clone/mw/master as front-end path (assuming that is correct) [15:21:27] clone/gerrit maybe, but that's just naming. Ah, okay, so it does include changePath at the end [15:21:56] it looked like it didn't since $wgScriptPath = "${testswarm.URL.path ; testswarm.URL.path=/clone/mw/master [15:22:13] ah testswarm.URL.path="${testswarm.URL.basepath}/${changePath} [15:22:20] gotcha [15:23:36] http://integration.mediawiki.org/clone/mw/master/test.txt wee [15:24:14] New patchset: Hashar; "testswarm: restore /etc/testswarm" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4787 [15:24:16] mutante: ^^^^ [15:24:27] oh? why [15:24:30] Project does not begin with mediawiki/ [15:24:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4787 [15:24:51] mutante: we do not want puppet to delete /etc/testswarm since the Debian package put its stuff there [15:25:01] mutante: so I am just reverting the change to ensure=>absent [15:25:22] "Project does not begin with mediawiki/" [15:25:27] hashar: got that part, ok good.thought for a moment you deleted stuff you still needed [15:28:24] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4787 [15:28:27] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4787 [15:29:20] k my Facebook friends are not geek [15:29:28] only two persons recognized: git push gerrit master:refs/for/master [15:29:36] heh [15:29:47] hashar: you need better friends :) [15:29:53] NOOOO [15:30:16] the good point otherwise is that when I invite them at home, we talk about everything but computer science :D [15:30:46] do you ever invite non-geeks and geeks on the same night? [15:30:46] <^demon> Other than my wmf friends I don't know anyone on facebook who'd get it. [15:31:40] my geek environment eventually grown up, got married and have childs :-D [15:31:43] at least one other who'd get it is anti-Facebook [15:32:02] so yeah, sometime we wine about how Everquest and RPG nights were the good old time [15:32:05] but that is about it [15:32:17] the main subject nowadays are diapers, school and wine :-] [15:33:35] nothing but info and notice in gallium puppet run now. and done after 60 seconds [15:34:34] \O/ [15:35:03] * hashar offers a virtual round of "alcohol free" beer [15:35:34] takes one [15:38:55] quick question: how do i find the hostname of the mysql db when i want to connect a specific wiki project? [15:38:59] <^demon> This beer is woefully unsatisfying :\ [15:39:57] <^demon> declerambaul: The db clusters are defined in db.php [15:40:02] <^demon> noc.wikimedia.org/conf/ [15:40:33]