[00:00:28] !log updating OpenStackManager to r114724 on virt0 [00:00:30] Logged the message, Master [00:02:24] hm [00:02:30] well, that surely isn't working [00:05:33] bah. fucking live hacks [00:05:37] I never checked that in [00:12:18] !log updating OpenStackManager to r114726 on virt0 [00:12:20] Logged the message, Master [00:19:31] !log updating OpenStackManager to r114728 on virt0 [00:19:33] Logged the message, Master [00:24:52] !log updating OpenStackManager to r114729 on virt0 [00:24:54] Logged the message, Master [00:33:36] !log updating OpenStackManager to r114730 on virt0 [00:33:38] Logged the message, Master [00:36:50] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [00:46:53] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [00:46:53] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [01:01:53] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [01:01:53] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [02:03:18] PROBLEM - udp2log processes on locke is CRITICAL: CRITICAL: filters absent: /a/squid/fundraising/bi-filter, [02:05:24] RECOVERY - udp2log processes on locke is OK: OK: all filters present [02:18:09] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [02:35:06] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [02:40:03] PROBLEM - Host lvs5 is DOWN: PING CRITICAL - Packet loss = 100% [02:43:21] PROBLEM - BGP status on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, sessions up: 7, down: 1, shutdown: 0BRPeering with AS64600 not established - BR [03:35:46] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Thu Apr 5 03:35:20 UTC 2012 [03:46:25] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [04:42:22] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [05:51:14] PROBLEM - MySQL Slave Delay on db24 is CRITICAL: CRIT replication delay 202 seconds [05:51:23] PROBLEM - MySQL Replication Heartbeat on db24 is CRITICAL: CRIT replication delay 206 seconds [05:59:51] RECOVERY - MySQL Replication Heartbeat on db24 is OK: OK replication delay 0 seconds [06:00:09] RECOVERY - MySQL Slave Delay on db24 is OK: OK replication delay 0 seconds [09:19:51] New review: Hashar; "Needs a few more tweaks." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/3285 [10:35:05] I need a sysadmin that can do some tweaks to OTRS - any suggestions? :) [10:35:13] eeuurghh [10:35:27] so I thought otrs was in the middle of being shuffled around. maybe it's just being talked about [10:35:34] anyways robh I think knows about this [10:35:39] and maybe [10:35:42] * apergos tries to remember [10:35:44] mutante? [10:35:50] what tweaks do you need? [10:36:10] so there's a 1 click spam button apergos on queue view that will move things to the "Junk" queue [10:36:14] yes [10:36:24] I'd like that to be duplicated for "1 click junk" to move to Junk (non spam) [10:36:25] (I have otrs queue from being a volunteer ;-) ) [10:36:35] ah [10:36:45] geez I have no idea how that stuff works. whatsoever [10:36:54] and also if possible, put both of those buttons in the message view - it's currently only in the queue view [10:37:06] but getting a button in queue view is more important than the message view one at the moment ;) [10:37:58] right [10:38:41] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [10:39:25] there's a whole list of improvements at https://otrs-wiki.wikimedia.org/wiki/OTRS_technical_challenges apergos - but I think the 1 click spam is probably going to be the most useful [10:40:49] ah, this is not going to be a five minute task [10:40:50] this is [10:40:56] make a patch similar to [10:40:58] http://svn.wikimedia.org/svnroot/mediawiki/trunk/otrs/patches/50-one-click-spam.patch [10:41:03] build and test package [10:41:04] dpleoy [10:41:27] but in the back of my mind I think that there is a migration to a newer version or a different platform or something in the works [10:41:33] * apergos wishes they had a memory that didn't suck [10:42:06] whenone of rob or mutante shows up we can find out [10:44:09] it would be nice to get enhancement requests like these somewhere a bit more public [10:44:51] there's a component for it in bugzilla [10:45:02] guillom: did you think that the upgrade's been stalled until later this year? [10:45:33] I bet that in fact no developer pays any attention to that page [10:46:09] I know Jeff or someone else was working with an OTRS expert to upgrade to 3.0, but I don't know if the upgrade has been explictly postponed [10:46:13] https://bugzilla.wikimedia.org/buglist.cgi?query_format=advanced&list_id=105276&component=OTRS&resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=LATER&resolution=DUPLICATE&resolution=WORKSFORME&product=Wikimedia [10:46:19] these need a walkthrough someday too [10:46:27] I guess they aren't all open, just a sec [10:47:10] https://bugzilla.wikimedia.org/buglist.cgi?query_format=advanced&list_id=105278&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=VERIFIED&component=OTRS&resolution=---&product=Wikimedia [10:47:13] much shorter list [10:47:36] so maybe adding the top few enhancements to the list would be good [10:48:13] apergos: FYI: https://rt.wikimedia.org/Ticket/Display.html?id=452 [10:48:35] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [10:48:35] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [10:49:03] ok, so at least it's somewhere in the pipeline [10:49:31] legal and/or philippe should probably be pinged again, I expect jeff knows what is going on [10:49:58] heh both my guesses about right people were wrong [10:49:59] I believe I'm right in saying Philippe met the OTRS guy in Berlin to discuss it [10:50:04] Thehelpfulone: so apparently they're still working on it, but waiting for an NDA to be signed with the volunteer OTRS expert who's helping us [10:50:17] that was done a while back guillom, apparently [10:50:18] Ah, I haven't heard about that [10:50:43] grrr [10:50:49] if only there were public rt queues [10:51:06] so what I was looking at and you can 't see Thehelpfulone is discussion about the nda [10:51:08] according to philippe, it was in the week beginning Monday 13th February [10:51:17] last update is Jeff_Green saying the process is stalled, as of mid march [10:51:35] so I would check in with him and see what's going on [10:52:01] okay [10:52:06] heh, break down in communication! [10:52:23] he is a us person so it will be some hours yet before he's online [10:52:46] yep [10:52:50] those americans! ;) [10:52:54] hah [10:53:05] I'm a "those americans", I'm just in a different timezone :-D [10:53:38] ok well I know those were not the tweaks you were looking for, [10:54:03] but anyways hopefully you'll be able to get them scheduled [10:54:29] yep, I'll look through that huge list to see what's most important [10:54:38] ok cool [11:00:57] guillom: is it okay to copy/paste those bugs from https://otrs-wiki.wikimedia.org/wiki/OTRS_technical_challenges - if I remove who said it? [11:01:32] probably; use your judgment :) [11:02:01] will do :) [11:03:44] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [11:03:44] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [12:20:08] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [12:27:24] !log search1 and search4 seem to be dead. restarting lsearchd [12:27:28] Logged the message, notpeter [12:29:38] bah, I did not see that or I would hav erestarted them [12:29:40] I'm sorry [12:30:05] apergos: it's ok :) [12:30:25] they weren't alerting in nagios, because theonly check we have is a tcp 8123 port check [12:30:32] (imporving this is on my todo list) [12:30:33] I got one of the search hosts yesterday, I forget which one [12:30:42] awesome! [12:30:44] but only cause I happened to see it scroll by in here [12:30:51] but yeah, tcp 8123 up! serving no traffic :/ [12:30:56] ok yeah ugh [12:31:16] yeah, I only notice because I start every day by looking at ganglia at this point :) [12:31:27] ;-) [12:31:39] see my warnings come via other channels so I loko at those instead :-D [12:32:44] oh, you mean your other channels actually tell you when there are problems, instead of having to divine them from graphs? [12:32:47] some day.... [12:32:50] I too will be like that [12:34:10] I mean I get emails [12:34:19] RobH: just checking in on ssds for eqiad search? [12:34:24] when things fail it's either a cron job or a job that emails me directly [12:34:33] ah, gotcha [12:35:02] the other thing I do is look at open screen sessions on a couple hosts where I have stuff half done and no notification system yet [12:35:06] (things in testing or whatever) [12:35:27] oh this stuff about new slaves is external cluster [12:35:32] I see [12:35:46] * apergos is reading up on yesterday's irc log and going to get one ready [12:37:01] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [12:37:28] I dunno why I thought it was some other type of slave :-D [12:38:11] we're not *that* kind of imperialist institution. we're the good kind! I promise! [12:38:30] riiiggghhht [12:38:56] when have white males ever led people astray on this point? [12:38:57] oh. [12:38:59] wait.... [12:40:54] I see, I'm going to get up to the rsync of the snapshot and then come back in two days [12:40:55] fine [12:43:39] Jeff_Green: you around? [12:43:53] yes, just arriving [12:44:15] sweet. I was gonna say "let the testing continue!" [12:44:24] sounds good [12:44:45] I'm going to try to get back to working on pediapress stuff today, but I can help out in parallel [12:44:59] Jeff_Green: sweet [12:45:19] notpeter: so some came in, but not all of them, and the brackets are in route [12:45:40] i will chase down whats goin on with them later today [12:46:09] my basic plan is: retest pool 2/3, just to be safe, point them at eqiad, wait an hour or so, test en, point at eqiad, then prefix host for en, then *, then *.prefix [12:46:15] ok [12:46:31] so if I can just poke you for occasional testing and sanity checking, that would be awesome [12:46:38] sure [12:46:51] RobH: awesome! rough eta until installed? a week? [12:47:05] why don't we just do some full-rig sweep tests and see how it goes? it's not that much more painful than a single-pool run [12:47:16] sure, sounds good! [12:47:16] dunno until i track them down, but whatever day they arrive, i will install them that day or the following ;] [12:47:26] i would hpe less than a week unless they fubar'd the shipment [12:47:32] RobH: great! mostly just curious [12:47:37] kk, cool [12:50:05] notpeter: starting /opt/searchqa/bin/api_sweep_test -t 10 -l -m 100 [12:50:11] sweet [13:11:51] Jeff_Green: those results look good. [13:12:00] the fails look like mostly timeouts in pmtpa [13:12:06] what are the percentages? [13:12:19] it's going to be a while [13:12:42] it's running only 10 threads to keep things mellow, it's about 20% complete [13:12:57] oh! [13:12:57] ok [13:12:57] although lemme see, I can get you a mid-run status now that I think of it [13:13:06] nah, that's cool [13:13:50] the only things I'm seeing are a lot of 500's on 10.2.1.12 [13:13:58] but that's pmtpa just being failful right? [13:14:28] some 500s from 10.2.1.13 too [13:14:45] apart from that things look good [13:18:44] I think based on the early results I wouldn't hesitate to enfire the live testing [13:20:37] sweet. I'm going to dump in pool2 and 3 now, as those were known good yesterday and don't look like they've died overnight [13:20:46] k [13:21:50] !log pointing de, fr, ja, es, ru, nl, pl, pt, zh, and sv search at eqiad [13:21:52] Logged the message, notpeter [13:25:47] apergos? es1004 and es1002 are being reslaved. rsync still runnning [13:25:55] uh huh [13:26:05] I'm just starting to set up the snap for sn1003 [13:26:08] es1003 [13:26:14] k [13:26:35] hashar: i'm here .. what was it that should be done on gallium [13:27:24] ton of stuff :-D [13:27:43] oh.. [13:29:46] apergos: oh, about OTRS tweaks, i dont really know [13:29:57] sok jeff should know [13:30:17] alright [13:31:21] notpeter: final results are in [13:31:36] i'll post 'em [13:32:21] hashar: how about i start with a bunch of package upgrades ?;9 [13:32:33] from ubuntu? sure go ahead [13:32:37] yea [13:32:45] notpeter: http://greenspoons.com/sqa_results.txt [13:33:28] !log installing package upgrades on gallium. apache,apt,postgres,php5-*,ruby,...various libs [13:33:29] Logged the message, Master [13:36:04] hashar: postgre AND mysql on same host? [13:36:11] yes [13:36:14] ok [13:36:15] Jeff_Green: any idea why pool3 was so awful? [13:36:16] under my responsability [13:36:31] though I need to poke ops from time to time since I am not root there :-] [13:36:35] pool3 at pmtpa? it just kept timing out [13:36:44] <^demon> mutante: Just fyi, those aren't databases that are stored and need backing up. They're just created on the fly for testing. [13:36:56] hashar: ok, what else did you need [13:37:01] ok, ^demon [13:37:08] maybe there's a crapped-out host at pmtpa? I havne't looked closer [13:37:13] Jeff_Green: oh! that would explain the low match rate... [13:37:18] mutante: phpunit upgrade. Let me find the RT ticket [13:37:32] ah, yeah [13:37:37] Jeff_Green: yeah, I'm a bit worried about search7, tbh [13:37:43] on zhwiki, svwiki, ruwiki, plwiki etc? [13:37:48] mutante: RT 2737 https://rt.wikimedia.org/Ticket/Display.html?id=2737 [13:38:06] yeah [13:38:15] that would also explain the long response time [13:38:22] mutante: I think PHPUnit got installed using the Ubuntu package which provide an outdated version of PHPUnit [13:38:43] notpeter: the poor scores were due to timeouts at pmtpa, so as long as the indexes on disk are fresh at eqiad I have far more faith in that side [13:38:56] yeah [13:39:06] mutante: instead we want to use php PEAR to download the latest PHPUnit and thus bypass Ubuntu package :D [13:39:11] I'm going to restart lsearchd on search7 anyway, while it's not getting traffic [13:40:35] k [13:41:07] and 96 match on en seems reasonable [13:41:23] like, the same up to some minor size differences and such [13:41:30] yeah totally, it's all doc size differences so I'm sure it's just due to index timing [13:41:43] yep! [13:41:47] hashar: there has been discussion about installing stuff from PEAR, like it is a third-party repo. But i'll upgrade it based on the fact that it does NOT appear like it was installed from the Ubuntu package before anyways and via pear in the first place [13:42:12] mutante: it was installed with ubuntu package [13:42:27] there is no package phpunit installed though [13:42:33] mutante: but I am not willing to spend two days backporting the PHPUnit package from a recent ubuntu distribution :-D [13:42:36] ohhh [13:43:02] so maybe it was installed with pear :-)))))))))) [13:43:26] !log gallium - upgrading pear [13:43:28] Logged the message, Master [13:43:31] yeahhh [13:44:13] <^demon> We installed it from pear in the beginning. The ubuntu repo copy is always woefully out of date. [13:44:19] <^demon> *sigh* package maintainers. [13:44:31] lets drop ubuntu and use LFS [13:44:50] uff [13:45:16] Jeff_Green: ?? [13:45:38] reponding to the upvotes for LFS and bleeding edge [13:45:42] !log gallium - upgraded phpunit and php_codesniffer via pear (have been installed via pear before, distro outdated) [13:45:44] Logged the message, Master [13:45:46] heh [13:45:51] hashar: PHP_CodeSniffer-1.3.3.tgz [13:46:04] mutante: phpunit --version : PHP Fatal error: Call to undefined method PHP_CodeCoverage_Filter::getInstance() in /usr/bin/phpunit on line 39 [13:46:12] * Jeff_Green likes not having to think about the status of package we rely on all day every day,and thanks the ubuntu folks for doing that for us [13:46:16] <^demon> Jeff_Green: I'm not asking for bleeding edge here, just latest stable that ubuntu fails to ship. [13:46:40] they do do that, it's true [13:46:41] ^demon: be glad we're not running debian :-P [13:46:46] mutante: I think we need to have everything updated [13:46:48] sometimes for years! [13:47:08] mutante: so just "pear upgrade" [13:47:29] Jeff_Green: instead we get to wait for the debian maintainers to make a package and for that to get ported to ubuntu! :) [13:47:30] <^demon> Jeff_Green: And there's a difference between running the bleeding edge of everything and upgrading carefully when we need new features. [13:47:35] isn't this where someone is supposed to insert a snide remark about volunteering to integration-test the latest stable? [13:47:44] <^demon> In this case, we actually need 3.6 and there's no way to get it other than PEAR or installing it manually. [13:48:01] you guys could make your own package ;] [13:48:09] OH NO PLEASE [13:48:13] not it! [13:48:16] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [13:48:42] I would be glad to provide packages if someone find me a nice tutorial / script to easily backport packages from latest ubuntu to whatever version we run currently [13:48:44] and when it's made and we introduce it to the rig, we have to take ownership of maintaining it as long as we use it [13:48:53] !log gallium - upgraded all pear packages [13:48:55] Logged the message, Master [13:49:08] <^demon> Gee, I'd love to write a package. But see this git migration has been keeping me awfully busy. [13:49:13] mutante: great!!! let me run a test [13:49:27] i was just stirring up shit to do it guys ;] [13:49:35] heheh me to, sorta [13:49:50] ;))) [13:50:16] <^demon> All this being said...PEAR is an awful package management system and I wish we didn't have to use it. [13:50:24] mutante: test running https://integration.mediawiki.org/ci/job/MediaWiki-GIT-Fetching/395/console [13:50:31] would it be possible to avoid needing 3.6? [13:50:32] and i was about to say "pear provider for puppet" :P [13:50:41] ^demon: lets migrate from PHP to node.js :-D [13:51:08] <^demon> Jeff_Green: There's a couple of new features we want in 3.6--qchris is writing some dump-related tests that take advantage of it. [13:51:22] <^demon> Testing of command-line scripts, for one. [13:51:32] i see [13:51:42] Jeff_Green: we need PHPUnit 3.6 to be able to test output of command line script :-( [13:51:49] 3.5 does not have anything to handle that [13:52:20] notpeter: i think your drives may have arrived i am going into eqiad now to check it out [13:52:28] will let you know shortly ;] [13:52:31] RobH: woo woo! [13:52:44] mutante: I think you can close https://rt.wikimedia.org/Ticket/Display.html?id=2737 now :) [13:53:00] maybe it's available prepackaged from one of backport repos? [13:53:11] if that's the case we could fetch and drop it in our own repo [13:54:54] that is somehow what I said before. We could be backporting Ubuntu packages [13:55:11] ah, i missed backscroll [13:55:12] I have not found any tutorial to do so though :-/ [13:55:26] and overall, I prefer having someone to su; pear upgrade; [13:56:27] that is faster than finding the backport, build it in labs, having the debs sent to subversion, harass someone to publish the package on apt, update puppet, have it merged in production then puppet ran :-] [13:56:31] * Jeff_Green goes to read backscroll [13:56:51] hashar: ok, done [13:56:59] it is totally worth it for the main cluster, but probably not on gallium which is some kind of a special machine. [13:57:11] OR, we could use some PEAR/puppet integration [13:57:14] that would be great [13:57:22] something like: include pear:phpunit :-] [13:57:31] kill does not seem to be doing the trick for mysql on es1003. [13:57:31] but gallium being a labs instance isnt an option? [13:57:58] I hesitate to -9 it even if we are usin a snapshot etc [13:58:04] mutante: eventually we will probably move it to labs yes [13:58:14] apergos: maplebed suggested -9 and thats what i used then [13:58:32] but that was on es1004 which was broken anyways [13:58:45] <^demon> hashar: Even if we don't make it generic, having gallium's pear stuff puppetized should be done. [13:58:53] hashar: moving it to labs sounds like a good way to avoid the third-party repo discussion.. at least until now [13:59:17] mutante: wanna upgrade jenkins ? :D [13:59:22] RT is https://rt.wikimedia.org/Ticket/Display.html?id=2041 [13:59:31] ^demon: https://gist.github.com/305778 [14:00:10] jenkins is build using an apt package though [14:00:19] !log pointing enwiki and enwiki.prefix at eqiad search cluster [14:00:21] http://apt.wikimedia.org/wikimedia/pool/universe/j/jenkins/ [14:00:21] Logged the message, notpeter [14:00:47] hashar: jenkins is already the newest version [14:00:53] <^demon> mutante: Oooh :) [14:01:32] FUD!!!!!!!!! [14:01:36] hashar: ok, reading ticket first [14:01:41] :))) [14:01:59] on gallium it uses apt.wm.org which has an outdated package [14:01:59] http://apt.wikimedia.org/wikimedia/pool/universe/j/jenkins/ [14:02:11] so we need to have that .deb updated from upstream [14:02:32] 1.458 currently [14:02:42] I have NO idea how that deb is built though [14:02:55] huh [14:03:05] mysqladmin shutdown did it when kill wouldn't [14:03:07] who knows [14:03:08] most likely it's just a rebuild of an existing ubuntu or debian package with no changes [14:03:36] <^demon> hashar: Who did that last time? [14:03:40] mark: it looks like it is just a copy of upstream deb package [14:03:45] ^demon: was going to ask you :-D [14:03:59] <^demon> I can't remember. [14:04:07] well... that looks fine [14:04:58] !log created labs account for cneubauer [14:05:00] Logged the message, Master [14:06:43] any reason I would get logged out of wikipedia sevrel times today? [14:06:55] PROBLEM - MySQL slave status on es1003 is CRITICAL: CRITICAL: Lost connection to MySQL server at reading initial communication packet, system error: 111 [14:07:02] hold on hashar, i think we just pushed the existing package as suggested [14:08:43] PROBLEM - MySQL replication status on es1003 is CRITICAL: (Return code of 255 is out of bounds) [14:15:55] mutante: so I have downloaded the jenkins package from upstream AND from apt.wm.org [14:16:01] mutante: they both have the same md5 sum [14:16:34] so I guess we should just copy the latest .deb ( http://pkg.jenkins-ci.org/debian/binary/jenkins_1.458_all.deb ) into http://apt.wikimedia.org/wikimedia/pool/universe/j/jenkins/ [14:16:58] it should be rebuilt first [14:28:47] mark: even if it is for "all" platforms anyways? i found the old ticket meanwhile, it was just imported with "reprepro -C universe includedeb lucid-wikimedia" [14:29:54] ah, it's a script? [14:29:57] then it may not be necessary [14:30:22] yea, i think this is why we ended up just doing it that way last time [14:30:28] php? [14:30:36] java [14:30:43] it is a Java .war [14:30:56] ah right [14:31:04] ok, import it then [14:31:08] yea, that .war file, and only that, i remember now [14:31:11] kk [14:32:28] hhhmmm, searches for "Bhkui wsxbdfdfv'dsfvsrfvdfv.slvsbkfdsv jlvfsd. Ivsfm,hkexfju dwf6!)..).):):):):7374hfflflfdhfjfjfjf$;$;$;$;$;&;&.$4jendncd" are failing on the new enwiki search infrastructure. but I think I'm ok with that ;) [14:33:04] I mean, I know that that cat walking across a keyboard has as much of a right to search on wikipedia as anyone else... but... [14:33:38] <^demon> Searching for that should easter egg to searching for "Cat on keyboard" [14:33:50] heh [14:33:56] as long as there's an article with that title! [14:34:01] !log importing jenkins_1.458_all.deb to wikipedia apt repo and upgrading it on gallium [14:34:02] Logged the message, Master [14:34:13] that would be a sweet error handler [14:34:16] hashar, ^demon , do you want to keep or overwrite your jenkins config? [14:34:30] <^demon> Keep, I assume? [14:34:37] /etc/default/jenkins that is [14:35:05] the diff is: AJP_PORT=-1 and PREFIX=/jenkins [14:35:13] the PREFIX was for that redirect afair [14:35:16] ok, keeping [14:35:48] oh, yea, JENKINS_ARGS also includes --prefix=/ci [14:36:06] <^demon> Yeah, we definitely need that one in JENKINS_ARGS [14:36:17] /ci is indeed our prefix [14:36:18] its upgraded [14:36:31] URL being https://integration.mediawiki.org/ci/ [14:36:43] <^demon> Stacktraces, whee [14:36:48] yep, remember we switched that back and forth in teh beginning [14:38:52] at hudson.plugins.git.GitTool.onLoaded(GitTool.java:74) ... sigh? [14:39:11] looking at /var/log/jenkins [14:39:19] there must be a faulty plugin that need an update [14:41:21] <^demon> Do we know what plugin is busted? [14:41:31] should be listed on , but not loading http://gallium.wikimedia.org:8080/pluginManager/installed [14:42:48] <^demon> :8080 is denied except via localhost. [14:43:19] ^demon: which plugin, i think "GitTool" or something, given the message at hudson.plugins.git.GitTool.onLoaded(GitTool.java:74) [14:43:34] <^demon> I'm trying to figure out how to disable it manually. [14:43:36] I think too [14:43:49] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [14:43:56] bleh, wtf is up with my nick not being available. [14:44:43] ^demon: wild guess: replace "install" with "remove" or similar in: "java -jar jenkins-cli.jar -s http://localhost:8080 install-plugin" [14:44:49] Rob_H: I'm squatting on it [14:44:52] no, but really [14:45:08] its some server side timeout [14:45:16] i guess someone tried to use it a bunch and its locked for awhile [14:45:20] <^demon> Yeah, I'm trying to figure out how to disable the plugin so we can at least get in, then update. [14:45:22] cuz its not online to ghost it [14:45:23] uh, I know that I said "feel free to turn off any search server you like" but, just to be extra super clear, that is not the case today [14:45:45] notpeter: thats directed at me i imagine? [14:45:48] yes [14:45:51] heh [14:45:54] I figured you knew this [14:45:59] well, when would be best to do the ssds? [14:46:05] but..... I really don't want to miscommunicate about it [14:46:08] why did you guys push eqiad live ;p [14:46:14] tomorrow? [14:46:17] we're testing [14:46:21] im not coming down here two days in a row ;] [14:46:23] so monday [14:46:28] sure, sounds good [14:46:36] ^demon: "After stopping Hudson/Jenkins, go to your HUDSON_HOME/plugins directory and remove both the .hpi file and the folder with the same name. " [14:46:37] thank you! [14:46:38] mutante: ^demon I think I found out how to disable plugin [14:46:41] ah are you pounding on ciscos today? [14:46:54] only when mark is around to assist [14:46:57] its blocked until then. [14:47:10] mutante: ^demon: touch /var/lib/jenkins/plugins/git.hpi.disabled [14:47:12] it seems silly for me to work with cisco when i have not had someone with more experience than me atleast take a look at it. [14:47:36] mutante: ^demon then I had sudo /etc/init.d/jenkins restart [14:47:55] i had to come onsite because we have a bunch of boxes in shipping [14:48:10] mmm new toys? [14:48:10] hashar: ^demon i did that, and now its down in a different way [14:48:17] and if i dont get them, they will either charge us to store them, or charge us to deliver to cage [14:48:28] and both of those come out to about my take home pay for a day =P [14:49:22] hashar: ^demon , check again :) [14:49:24] mutante: well I had it working at one point [14:49:41] yeahhhhhh [14:49:43] \O/ [14:49:47] hashar: it works now, we just started/restarted at the same time or so, we should stop working on it the same time :) [14:49:52] going to run the plugin upgrade now [14:51:06] !log gallium - disabled incompatible GitTool plugin on jenkins and restarted it [14:51:08]