[00:06:57] ops, my buddies! [00:07:02] can you approve this one pretty please? [00:07:13] https://gerrit.wikimedia.org/r/#/c/12299/ [00:07:25] mr. maplebed can help me maybe? [00:08:07] ottomata: do we have ct approval ? [00:10:49] !log stopped puppet on cp1020 until tomorrow - testing new build of the squid mobile redirector on one server until tomorrow [00:10:54] Logged the message, Master [00:10:56] preilly: ^^ [00:11:08] binasher: SWEET [00:13:21] does robla approval != CT approval? [00:13:29] does not [00:13:33] iiiinteresting [00:13:37] I will tell diederik that for the future [00:13:40] mr. woosters? [00:13:41] cool [00:13:50] yes mr ottoman [00:13:53] do you know about mr. Evan Rosen wanting access to stat1 and some other stuff? [00:13:59] Diederik told me to create an account for him [00:14:16] https://rt.wikimedia.org/Ticket/Display.html?id=3119 [00:14:23] yes, i have asked notpeter to look at it tomorrow morning when he gets online [00:14:34] ok, i've already done puppetization of everything [00:14:38] except for internproxy [00:14:46] cause I don't know what that is, and I dont' see it in puppet [00:14:49] so it just needs approved/merged [00:15:18] i can merge it if i get your ok woosters (or we can wait until morn) [00:15:30] that is ok [00:15:40] but still need the acct setup in internproxy [00:16:20] leslie - if asher is there, he could tell u about internproxy [00:16:27] ah [00:16:54] ottomata: i was about to run to the gym, mind if we get this tomorrow (aka notpeter probably getting it) [00:17:51] LeslieCarr: its a thing that Ryan_Lane built :D [00:18:55] then definitely too much for a quick 30 second explanation ;) [00:19:05] yeah no probs [00:19:18] i think evan doesn't need it today [00:19:37] hey binasher, who else should I get to approve my mysql /var/run vs. /run stuff? [00:19:44] if you and mark both +1 ed it? [00:20:01] ottomata: do you have rights to merge it yourself? [00:20:35] New review: Lcarr; "Plus 1 pending internproxy setup" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/12299 [00:20:47] i'm off :) see you later [00:21:04] no [00:21:07] i don't have any merge powers [00:21:25] wouldn't it be nice if I did though? [00:21:28] oh yes indeedy [00:22:19] hah [00:22:39] i'll merge it for you [00:22:44] binasher: what did I build? [00:22:55] oh [00:22:58] internproxy? heh [00:23:05] I really want tesla to die [00:23:14] i blame… the analinterns [00:23:26] yay, thank you binasher [00:23:38] lemme know when it is on puppetmaster and I will run it on stat1 to see it do good [00:23:45] agreed [00:23:48] it shouldn't affect anywhere else [00:24:40] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11296 [00:26:00] gah [00:26:01] Your change could not be merged due to a path conflict. [00:26:02] Please merge (or rebase) the change locally and upload the resolution for review. [00:38:58] ottomata: i'm about to get picked up from the office and won't be back on for a few hours. could you do the rebase? [00:47:58] brwwwwww [00:48:05] yeah, i will do it tomorrow [00:48:10] ergh, eating dinner now [00:48:14] path conflict [00:48:15] hmm [00:59:42] New patchset: Faidon; "Add virt6-8 to netboot.cfg" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12308 [01:00:14] New review: Faidon; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12308 [01:00:14] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12308 [01:00:20] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12308 [01:02:07] New patchset: Faidon; "Add virt6-8 to site.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12309 [01:02:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12309 [01:03:12] New review: Faidon; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12309 [01:03:15] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12309 [01:49:02] New patchset: Faidon; "partman: use sda-sdh for Ciscos instead of sdc-sdj" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12316 [01:49:36] New review: Faidon; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12316 [01:49:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/12316 [01:53:30] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12316 [05:30:44] !log clearing mobile varnish cache - my friend can't expand some article categories on his iphone after rebooting and clearing cache [05:30:50] Logged the message, Master [07:48:14] what you guys think of providing a server to freenode to host their ircd as some kind of long term donation [09:11:54] petan, they only have to ask a decommissioned server, not hard [09:12:47] they can't have a server in the WMF datacentre because they want full access to it, basically a dedicated server they own, and this is not possible [09:13:28] Nemo_bis: it could be a virtual server on labs [09:13:33] that would be secure [09:13:49] or it could be a decommisioned server which would be firewalled [09:14:01] decommissioned servers go out [09:14:04] also they don't want full access, they need non root access to ssh to user who can run ircd [09:14:16] need or want? [09:14:23] need to maintain ircd [09:15:56] "we greatly prefer dedicated machines with root access" [09:15:59] http://freenode.net/hosting_ircd.shtml [09:16:06] prefer, but not demand [09:16:26] anyway if it was on labs or such, they could have root anyway. [09:16:44] machines there are virtual but so powerfull it would be enough for ircd [09:17:08] anyway I still think that they should be fine with non root [09:17:14] I can ask them [09:17:24] but I wanted some input from ops first, if it's even possible [09:17:42] no, they're usually not fine with non root [09:17:56] Nemo_bis: I can ask them [09:18:08] question now is if it's even possible [09:18:27] I think it's better to investigate whether they're interested first [09:18:51] there is a little point in investigating that if it wouldn't be even possible [09:19:01] I would prefer to ask someone from ops first [09:19:10] * Nemo_bis shrugs [09:19:16] if there was some option, then I ask freenode [09:20:45] suggestions in the past to actively (= financially) helping FreeNode have not been very favourably received (hosting donations ~ financial help) [09:31:15] yes, I disagree with financial support but I like the idea of providing a server resources to freenode [09:31:24] it seems to be a better option [09:31:33] more transparent as well [09:31:44] you can never know where the money would go to [09:32:08] but providing the servers, given there are many of them we could use is no problem at all, it's cheap and useful for freenode [09:39:20] oh you mean like irc.wikimedia.org would be linked into freenode? [09:41:13] http://freenode.net/hosting_ircd.shtml [09:42:57] sorry, i obviously should have read more backlog first [09:46:12] agree to petan (server is better than financial) if we really want to give them root on something misc. that is totally separated from other stuff and we can take the network requirements [09:47:51] in this moment I have no idea if they really require to have a root [09:48:07] I doubt so [09:48:34] but question is if someone from operation team would support that idea of providing some resources to freenode as kind of donation [09:52:20] but you can talk them out of requiring root if you tell them its all puppetized and they can use gerrit [09:52:24] maybe [09:54:30] i would support the idea of donating some resources to them as we indeed heavily use their network. ask network people about the requirements and separation though and hardware people about availability of "misc" hardware with these specs (maybe separate tickets is best) [11:14:32] mutante: right [11:27:01] mutante: ok, is there any decomissioned server that could be used for this purpose? they would like to know some technical details [11:27:30] petan: please ask robh about that part [11:27:34] ok [11:28:45] mutante: what is his email? [11:30:03] petan: let me create a ticket for it instead, what is your email to be added to cc: [11:30:15] benapetr@gmail.com [11:30:25] a ticket? decommissioned servers are always handled all together [11:33:48] petan: you now have 3174@rt.wikimedia.org. feel free to mail summary of the talk above, the specs, the link etc directly to that? ok? [11:34:05] ok [11:34:17] Nemo_bis: there is more than one, they need specific specs, and just one.. needs the context what for etc.. [11:35:41] a server to use for what? [11:35:49] a freenode irc server [11:35:55] did you talk to mark about this? :) [11:36:14] no, i am trying to get it into tickets so everybody can reply :) [11:36:17] ah ok [11:36:59] I think I'm fixing the "No credentials for your account" bug right now [11:37:09] or I'm breaking the ldap extension. one of the two [11:37:21] but exactly yea, i did point out maybe its best to have 2 tickets, one for hardware donation and one for networking (can we separate, do we even want to) and access request or not [11:37:32] if this means that we're setting up privileged channels to get decommissioned servers, I don't like it (but it's p to Rob) [11:37:41] maybe it turns out that we _just_ donate servers but dont run anything on our network.. but maybe not [11:38:01] why would we donate the server? [11:38:06] if we are running it on our network [11:38:16] well, all of a sudden it was about "decommissioned servers" [11:38:21] * Ryan_Lane nods [11:38:22] which sounds like hardware donation [11:38:29] we decommission them because they are old [11:38:34] but before it was about actually running a server that fits their specs [11:38:43] and we don't want to deal with them anymore, because they might break [11:38:45] yea, another reason to clear that all up in tickets:) [11:38:49] yeah [11:39:16] Ryan_Lane: any ideas why the %$%$%!$@ Ciscos do not boot? [11:39:24] paravoid: no idea [11:39:25] Nemo_bis: in the past we have had (iirc) decommissioned servers we could not find homes for [11:39:29] and how fucking long does it take for them to boot [11:39:35] the one I build manually with the same config worked [11:39:37] that said.... [11:39:40] it took a few tries [11:39:47] because the raid had to build itselt [11:39:49] good luck with LDAP [11:39:53] and it timed out a few times [11:39:58] fucking software raid [11:40:11] paravoid: it takes ..forever [11:40:16] but eventually they do it [11:40:22] and yes, the ciscos take *forever* to boot [11:40:33] because it checks all of the ram every time [11:40:45] and with 185GB of ram, that's slow :) [11:41:22] it's not just that [11:41:23] apergos, not recently apparently https://blog.wikimedia.org/2011/06/13/server-decommission-donations/ [11:41:24] everything's slow [11:41:25] how could they think it was a good idea to not have a switch to skip the memory check in the bios?? [11:41:54] Nemo_bis: have to check with rob and mark, they will know about more recent decommisions [11:42:04] but I am pretty sure a batch went from esams without homes [11:43:19] petan: so there are several ways ,nothing at all / just donate old servers / just donate money / donate a server that is actually new and they can still use / run on on our network (with or without access for them/) etc. first needs somebody to sort out what different people were expecting. [11:43:24] Ryan_Lane: I have no idea what their hw requirements are... but I am sure that ircd's would run fine even on some virtual server [11:43:41] mutante: I am writing a mail to ticket [11:43:56] I'd prefer not to run that in labs :) [11:43:58] mutante: donating money doesn't sound like a good idea to me [11:43:58] the requirements are here: http://freenode.net/hosting_ircd.shtml [11:45:40] yep, already agreed on that. and thanks for writing it all as a mail [11:48:20] we have several donated servers in Tampa that we could send. older PE 1950's and the old SUN search boxes....I don't think they will support virtualization as they sit. [11:48:38] meant severals servers for donation [11:53:26] I don't think they accept the hardware as it is, but rather a servers that are already on some network and running, but I don't really know that [11:53:42] i guess a plus for us actually running a server would be having more control over limits when running into issues at hackathons etc [11:54:10] including possibility to have our bots running on local ircd server with no limits :) [11:54:17] nagios etc [11:54:23] which sometimes get killed for spamming [11:56:19] Ryan_Lane: it doesn't need to run on labs, but if it was running on virtual system you would probably have better control over the network, even if they had root access [11:56:42] I just wanted to say that these ircd are usually lightweight programs [11:57:02] "Our servers must be able to open at least 15,000 concurrent connections, so per-user and system-wide file descriptor limits must be high enough to accomodate this. Suggested values are at least 24,000 for the freenode user account, and 25,000 system-wide." [11:57:20] mutante: but these connections have low traffix [11:57:22] traffic [11:57:29] irc protocol is just a text [11:58:20] petan: what do you mean by "wiki projects" [11:58:26] which ones are rejected by freenode? [11:58:27] paravoid: where [11:58:54] wikitech [11:59:03] paravoid: oh this, that was some complaint by vargent, not me, I only know that he registered a channel on freenode and he was rejected because it's a wiki and these aren't allowed to be hosted on freenode [11:59:08] varnent [11:59:37] he talked to sumana regarding that, I don't know details [12:00:04] but he said that freenode told him that wikimedia is affiliated to mediawiki and that is open source, thus we can have channels here or something like that [12:00:42] I think he was generally talking about 3rd wiki projects using mediawiki as their sw [12:01:33] nearly positive I just fixed the no nova creds for your account problem [12:01:40] on my test wiki [12:01:42] Ryan_Lane: yay [12:01:55] :) [12:01:59] now to think about the consequences of the solution [12:02:03] Nemo_bis: about hardware donation of decom'ed servers in general. there are no privileged channels. once it has been announced publicly and people all sent their request for hardware into RT and this is just a ticket in the queue like the others [12:03:00] so, the only time that the domain needs to be re-fetched is when a login occurs from a token [12:03:04] the long-lived one [12:03:10] otherwise it can get the domain from the session [12:03:45] the token is only valid in one login-session at a time. if you login again, with the "remember me" option selected, it overwrites your token [12:03:52] and your old long-lived session is gone [12:04:16] so, I set the domain in the user's options (which gets stored in the database) as well as setting it for the session [12:04:28] hm. [12:04:33] that's not going to work [12:04:40] I need to only do it if the token is also set [12:05:06] otherwise a non-token login can modify a token-login's domain [12:16:52] not at all -> i guess a plus for us actually running a server would be having more control over limits when running into issues at hackathons etc [12:27:12] \o/ [12:27:14] fixed it [12:27:29] and did it a way that is actually secure [12:30:05] pretty sure I fixed another bug by doing this too :) [12:38:05] of course, I need to make core changes for this [12:38:58] yay \o/! [12:39:04] 111oneoneone [12:40:57] heh [12:41:28] I'm going to backport my changes into the version of mediawiki we're using on labs, of course [12:41:36] well, on labsconsole itself, anyway [12:42:02] I was waiting on this to enforce two-factor ;) [12:46:18] mutante: btw, what's the mailman admin password? [12:46:33] or, to make my question more clear [12:47:02] we are getting bounces to gerrit@ because apparently it mails mediawiki-commits and mediawiki-commits doesn't allow it because of 'implicit destination' [12:47:09] if freenode needs root on the system it's likely a non-starter [12:47:25] (I have to do something while I wait for the @#%$ Ciscos to boot) [12:47:29] heh [12:47:36] paravoid: feel my pain [12:47:50] I rebooted one of those fuckers about 50 times when we were trying to get PXE working [12:49:02] the thing is, I can't understand why Andrew's machine booted and mine isn't [12:49:32] * Ryan_Lane nods [12:51:28] I have little experience with raid10 md, but how can grub boot from them? [12:51:34] they're striped (too), aren't they? [12:51:56] the same way it would boot from a raid1 [12:52:13] grub has support for booting from a software raid, I believe [12:53:18] oh, grub2 does that [12:53:19] didn't know that [12:53:24] I've always avoided grub2 :) [12:55:10] the other extremely annoying about the ciscos is their mgmt timeout [12:58:10] Nemo_bis: oh , i got you wrong first then, i thought you worry about privileges for receiving used hardware originally bought from donations [12:59:27] mutante, no, I just worry about Rob having to manage many single-server donations processes [12:59:32] :) [12:59:54] afraid he has to but this is not a new thing at all [13:00:17] with all these requests for hardware from public that arrived like a giant batch job [13:01:30] paravoid: the timeout is annoying, but connecting to the console is even more annoying [13:01:35] because you can't disconnect [13:01:40] yes, that too! [13:01:43] goddammit [13:01:56] who in their right minds doesn't include an escape sequence? [13:35:12] New patchset: Dereckson; "(bug 37778) Enable LiquidThreads on fi.wikimedia.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/12366 [13:35:20] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/12366 [13:37:01] something's seriously wrong with this partioning [13:37:40] Ryan_Lane: btw, do you like /b? [13:37:41] I don't [13:38:09] I'd say md + LVM + ext4, one LV for / and perhaps another LV for instance storage [13:47:10] New review: Dereckson; "shellpolicy issue resolved" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/11843 [14:06:40] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=&s=by+name&c=Application%2520servers%2520pmtpa&tab=m&vn= [14:06:41] hmm [14:07:33] https://gdash.wikimedia.org/dashboards/reqerror/ [14:07:41] fcking nagios [14:08:02] well I did get a page jus tnow [14:08:27] srv272 load spiking [14:08:32] seems like yesterday's fault again [14:08:46] got a page, ack [14:09:18] and the one its OK again [14:09:28] yeah [14:09:33] paravoid: it was originally /a [14:09:44] I was looking at 272 (as in the logs, couldn't actually get on it) [14:09:47] till the giant gluster splitbrain [14:09:54] then I moved everything to /b [14:10:09] nothing too special in the apache logs though [14:10:10] paravoid: this is a good time to change it to something you like [14:10:27] this is a good time to have a look at the downtime... [14:10:30] dammit [14:10:35] you could just mount it at /var/lib/nova/instances [14:10:40] downtime? [14:11:13] ugh [14:11:30] what happened to ms-be5 btw [14:11:31] !log powercycling srv272, unreachable due to load spike [14:11:37] Logged the message, Master [14:11:38] maplebed: ? [14:11:39] it must have swapdeathed [14:11:45] http://ganglia.wikimedia.org/latest/?c=Application%20servers%20pmtpa&h=srv272.pmtpa.wmnet&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [14:11:50] memcache issues [14:12:08] lots and lots of memcache issues [14:12:40] just that one box [14:12:46] mark: same as yesterday [14:13:01] cache storm for one object? [14:13:27] where do you see the memcached being a problem? [14:13:28] I did that [14:13:52] PROBLEM - Apache HTTP on srv269 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:06] on that box [14:14:10] apergos: /home/w/logs/memcached.log [14:14:19] RECOVERY - Apache HTTP on mw42 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.838 second response time [14:14:24] "2012-06-21 14:14:13 srv259 plwiki: Error parsing memcached response" [14:14:28] RECOVERY - Apache HTTP on srv203 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.983 second response time [14:14:37] RECOVERY - Apache HTTP on srv236 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 1.605 second response time [14:14:46] RECOVERY - Apache HTTP on mw9 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.888 second response time [14:14:46] RECOVERY - Apache HTTP on mw19 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.690 second response time [14:14:55] RECOVERY - SSH on srv272 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:14:55] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.pmtpa.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 47321 bytes in 0.129 seconds [14:15:04] RECOVERY - LVS HTTP IPv4 on foundation-lb.pmtpa.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 38959 bytes in 0.147 seconds [14:15:04] RECOVERY - Apache HTTP on mw1 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.238 second response time [14:15:13] RECOVERY - Apache HTTP on mw10 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [14:15:13] RECOVERY - Apache HTTP on srv269 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.091 second response time [14:15:14] okay, the powercycle fixed it it seems [14:15:22] RECOVERY - Apache HTTP on mw7 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.162 second response time [14:15:22] ACKNOWLEDGEMENT - Host ms-be5 is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn RT #2595: investigate ms-be5 for hardware failure [14:15:22] RECOVERY - Apache HTTP on mw11 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.902 second response time [14:15:23] now let's look at atop to figure out why the fuck that machine died [14:15:24] annoying that one box can take us down [14:15:32] RECOVERY - Apache HTTP on mw16 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [14:15:32] RECOVERY - Apache HTTP on mw20 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.045 second response time [14:15:32] RECOVERY - Apache HTTP on mw26 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.455 second response time [14:15:32] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [14:15:36] Ryan_Lane: very [14:15:40] RECOVERY - Apache HTTP on mw43 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [14:15:40] RECOVERY - Apache HTTP on mw55 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [14:15:40] RECOVERY - Apache HTTP on mw5 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [14:15:40] RECOVERY - Apache HTTP on mw21 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [14:15:40] RECOVERY - Apache HTTP on mw24 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [14:15:41] RECOVERY - Apache HTTP on mw38 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [14:15:41] RECOVERY - Apache HTTP on mw49 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [14:15:42] RECOVERY - Apache HTTP on mw14 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.132 second response time [14:15:42] RECOVERY - Apache HTTP on mw37 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 1.789 second response time [14:15:43] RECOVERY - Apache HTTP on mw29 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.138 second response time [14:15:43] RECOVERY - Apache HTTP on mw52 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.314 second response time [14:15:49] RECOVERY - Apache HTTP on mw6 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [14:15:49] RECOVERY - Apache HTTP on mw32 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [14:15:49] RECOVERY - Apache HTTP on mw53 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [14:15:49] RECOVERY - Apache HTTP on mw41 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [14:15:49] RECOVERY - Apache HTTP on mw36 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [14:15:50] RECOVERY - Apache HTTP on mw34 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.041 second response time [14:15:50] RECOVERY - Apache HTTP on mw46 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.045 second response time [14:15:51] RECOVERY - Apache HTTP on mw25 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [14:15:51] RECOVERY - Apache HTTP on mw2 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.041 second response time [14:15:52] RECOVERY - Apache HTTP on mw4 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.192 second response time [14:15:52] RECOVERY - Apache HTTP on mw54 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.280 second response time [14:15:53] RECOVERY - Apache HTTP on mw3 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.046 second response time [14:15:53] RECOVERY - Apache HTTP on mw30 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [14:15:54] RECOVERY - Apache HTTP on mw17 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [14:15:54] RECOVERY - Apache HTTP on mw13 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [14:15:55] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.044 second response time [14:15:55] RECOVERY - Apache HTTP on mw44 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.381 second response time [14:15:56] RECOVERY - Apache HTTP on mw57 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.871 second response time [14:15:56] RECOVERY - Apache HTTP on mw51 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.328 second response time [14:15:57] RECOVERY - Apache HTTP on mw18 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.217 second response time [14:15:58] RECOVERY - Apache HTTP on mw45 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [14:15:58] RECOVERY - Apache HTTP on mw28 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.055 second response time [14:15:58] RECOVERY - Apache HTTP on mw47 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.039 second response time [14:15:59] RECOVERY - Apache HTTP on mw33 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [14:15:59] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.040 second response time [14:16:00] RECOVERY - Apache HTTP on mw48 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [14:16:00] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [14:16:01] RECOVERY - Apache HTTP on mw12 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.395 second response time [14:16:01] RECOVERY - Apache HTTP on mw40 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [14:16:02] RECOVERY - Apache HTTP on mw15 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [14:16:02] RECOVERY - Apache HTTP on mw22 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.053 second response time [14:16:03] RECOVERY - Apache HTTP on mw39 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.477 second response time [14:16:07] RECOVERY - Apache HTTP on mw56 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [14:16:12] hopes the bot isnt banned [14:16:16] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [14:16:27] I restarted the bot btw [14:17:01] RECOVERY - Apache HTTP on mw58 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.048 second response time [14:22:19] jobrunners & memcached on the same machine??? [14:22:21] W-T-F [14:22:30] maplebed: re: RT-2595 please select: (powercycle since still in rotation, take out of rotation and give up on it due to previous hardware issues [14:22:32] for a long time now [14:22:48] I thought we tried not to do that... [14:23:56] 7518 634667 0 7442K 218.8M 60800K 2388K 2796K 91% php [14:23:57] mutante: sorry, I forgot about that ticket. [14:23:59] 7517 607367 0 7442K 218.7M 60628K 1492K 1716K 91% php [14:24:02] 7516 34015 0 7442K 256.0M 97960K 24476K 24036K 86% php [14:24:06] that's the last minutes of srv272 [14:24:07] ms-be5 being down is intentional at the moment [14:24:08] maplebed: normal to get refused on mgmt? [14:24:13] (putting ssds in) [14:24:18] maplebed: alright, gotcha [14:24:23] 18315 14604 922 7442K 240.3M 69116K 19132K 6220K 30% php [14:24:26] 18316 43009 905 7442K 239.6M 68964K 18024K 4716K 29% php [14:24:29] 18313 94696 1110 7442K 244.2M 74892K 20664K 8596K 29% php [14:24:32] 18312 18207 1130 7442K 232.7M 65828K 10268K 1672K 28% php [14:24:35] 18314 22268 1093 7442K 275.3M 98.5M 56076K 37844K 27% php [14:24:38] 7516 134208 0 0K 0K 0K 0K 0K 100% [14:24:41] 7518 1011e3 0 0K 0K 0K 0K 0K 100% [14:24:44] 7517 960000 0 0K 0K 0K 0K 0K 100% [14:24:46] heh [14:25:08] paravoid: job runners and memcache should not be on the same machine [14:25:13] if they are, that's a misconfiguration [14:25:28] they are indeed [14:25:29]