[09:43:05] um [09:43:07] !ping [09:43:07] pong [09:47:36] !ping [09:47:37] pong [09:56:52] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 157 processes [10:36:32] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 27% free memory [10:37:22] PROBLEM Free ram is now: WARNING on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [11:02:22] RECOVERY Free ram is now: OK on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: OK: 20% free memory [11:10:22] PROBLEM Free ram is now: WARNING on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [11:10:41] Hey there! I have discovered a super power of mine: I can make the mysql db of a working MW not show in mysql! [11:10:48] :( [11:11:48] It's on a puppet-installed mediawiki: If I show the databases, I don't see it. Though the wiki is working. [11:12:32] What's even stranger: When I connect to that same mysql from a different labs instance via the network I can see it. [11:12:46] Silke_WMDE: hah, MySQL RedPill [11:13:18] looks like an some entries in the "mysql" database? host vs db tables? [11:14:02] show databases doesn't show it [11:14:21] while mysql -u root -p -h 10.4.0.23 -D blah shows it [11:15:46] I think I had the same last week, only then I assumed puppet had failed to create the db [11:15:56] but now I see the working wiki in fron tof me [12:38:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:40:32] RECOVERY Free ram is now: OK on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: OK: 21% free memory [12:42:32] PROBLEM Free ram is now: WARNING on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: Warning: 6% free memory [12:55:42] PROBLEM Free ram is now: WARNING on mw1-21beta-precise.pmtpa.wmflabs 10.4.0.174 output: Warning: 12% free memory [12:56:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [12:58:32] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 17% free memory [13:05:52] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [13:10:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:11:33] PROBLEM SSH is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Server answer: [13:15:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:16:32] RECOVERY SSH is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [13:56:22] PROBLEM Free ram is now: WARNING on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [13:57:31] Hi, who should I ask to join the project Bots? [13:58:20] Dakdada hi [13:58:22] me [13:58:37] what is your name [13:59:37] My pseudo is "Darkdadaah" [13:59:47] @labs-user Darkdadaah [13:59:47] That user is not a member of any project [14:00:00] is that your account on labs? [14:01:23] RECOVERY Free ram is now: OK on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: OK: 20% free memory [14:01:47] Yes it is. [14:02:07] I have shell access too (bastion). [14:02:17] !log bots +Darkdadaah [14:02:18] Logged the message, Master [14:02:21] ok [14:02:28] you are in project now [14:02:40] !botsdocs | Dakdada [14:02:40] Dakdada: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [14:03:24] Thank you :) [14:06:32] Oh, I think I know what was wrong: it's actually "darkdadaah" not "Darkdadaah" [14:06:59] @labs-user Darkdadaah [14:06:59] Darkdadaah is member of 1 projects: Bots, [14:08:04] @labs-user darkdadaah [14:08:04] That user is not a member of any project [14:09:16] Okay... well since I connect with "darkdadaah" I don't have access to anything else than Bastion. [14:09:55] you need to create a private key on bastion or use [14:10:00] !forwarding [14:10:00] https://labsconsole.wikimedia.org/wiki/Access#Using_agent_forwarding [14:11:19] oh ok thanks [14:15:52] PROBLEM Free ram is now: CRITICAL on mw1-21beta-precise.pmtpa.wmflabs 10.4.0.174 output: CHECK_NRPE: Socket timeout after 10 seconds. [14:20:43] RECOVERY Free ram is now: OK on mw1-21beta-precise.pmtpa.wmflabs 10.4.0.174 output: OK: 28% free memory [14:27:23] RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 21% free memory [14:30:22] PROBLEM Free ram is now: WARNING on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [14:50:16] I can now connect to the bots instances, but I need a directory for my files. I'd like to have a directory in /data/project/daahbot (this is the name of my bot) but I don't have the permssion to do so. [14:50:45] ok, sec [14:51:57] it's there [14:52:40] Perfect! Thanks! [15:05:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [15:06:23] PROBLEM Disk Space is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [15:06:33] PROBLEM dpkg-check is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [15:07:07] PROBLEM Total processes is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:10:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:11:22] PROBLEM Disk Space is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:11:22] PROBLEM Current Load is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:11:32] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:14:32] PROBLEM SSH is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Server answer: [15:15:54] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:16:24] RECOVERY Disk Space is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: DISK OK [15:16:24] RECOVERY Current Load is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK - load average: 0.10, 0.32, 0.38 [15:16:34] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [15:17:04] RECOVERY Total processes is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: PROCS OK: 179 processes [15:19:24] RECOVERY SSH is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [15:36:52] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 146 processes [16:04:37] Dakdada: Welcome! :) [16:05:11] Thank you Silke_WMDE :) [16:38:33] RECOVERY Free ram is now: OK on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: OK: 20% free memory [16:41:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:41:56] Does anyone know if there is a repository of projects dumps accessible for the bots? [16:45:56] Answer to myself: yes, there is: /public/datasets/public/ [16:46:32] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 17% free memory [16:59:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:04:24] PROBLEM Current Load is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:09:22] RECOVERY Current Load is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK - load average: 0.17, 0.18, 0.28 [17:11:15] andrewbogott: hey andrew :) [17:11:31] hey -- I see your request, still thinking about how to implement. [17:11:35] guess you are still away oh no :) [17:11:38] yeah so more wildcard [17:11:45] I don't know if we support wildcards in the middle of hostnames, I might just enumerate everything. [17:12:01] after a discussion with MaxSem, the proposed scheme of lang.project.m.beta.wmflabs.org does not match production so we need moaaar wildcard entries [17:12:09] and I haven't figured out how to add them in OpenStackManager :( [17:12:12] Oh, wait, nm, I misread. [17:12:14] Makes sense. [17:12:34] so just *m..beta.wmflabs.org [17:12:42] I don't think you can wildcard project in DNS [17:12:56] I think you might not have the privs. Can you see this page? https://labsconsole.wikimedia.org/wiki/Special:NovaDomain [17:13:19] I cant [17:13:23] must be a "cloudadmin" [17:13:36] so you have to create a new domain to get wildcards ? [17:13:53] Ryan should probably make you a cloudadmin :-D [17:13:56] Yeah, you create a domain and then create the host with name '*' under the domain [17:14:18] Anyway -- it's easy, I'll set things up shortly. [17:15:18] once that is up, we will have a beta mobile :-] [17:15:42] I still have to get mark to approve the changes I made to the role::cache::mobile and other varnish conf [17:15:49] but thanks to puppetmaster::self that is already on the instance [17:16:17] andrewbogott: the bug report listing the entries is https://bugzilla.wikimedia.org/show_bug.cgi?id=44669 [17:16:27] yep, I see it. [17:16:59] lamely copy pasted your explanation above :-) [17:20:22] PROBLEM Free ram is now: CRITICAL on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Critical: 4% free memory [17:24:09] hashar: OK, there's a bug somewhere in your system, it looks like it checks for unique domain name rather than unique FQDN which means it's only letting me having one m. [17:24:27] So I need to fix that before I can create your domains. So… there will be delays. [17:24:35] doh [17:24:56] * hashar setup a bind server to handle beta.wmflabs.org out of plain text files :-D [17:25:18] Ah -- I'm sure the backend can handle it, I think the web interface is just over-validating. [17:25:20] have you tried deleting the m.beta.wmflabs.org entry ? [17:25:46] no, but… we need that anyway, don't we? [17:25:54] * Damianz jumps up and down [17:26:05] you can drop it [17:26:12] m.beta.wmflabs.org is a scheme I made up [17:26:12] And I think it would still limit me to just one, whereas we need a bunch. [17:26:19] Damianz, do you have something to share with the class? [17:26:38] andrewbogott: I'm hungry, wanna make me food? [17:26:57] Oh, thought you had an idea about DNS, didn't realize you were just in a jumping mood :) [17:27:33] hashar: In the meantime do you want to pick one favorite domain and have me create that? [17:29:43] andrewbogott: Nope - our dns system suuuuucccckkkks ;) [17:29:49] Allways jumpy [17:32:00] andrewbogott: *.m.wikipedia.beta.wmflabs.org [17:32:08] andrewbogott: that would be a good start [17:32:13] 'k [17:32:56] ARGH [17:33:01] I forgot to ring someone :( [17:33:03] dammit [17:33:27] hashar: And it should point to deployment-varnish-t3 right? [17:33:49] andrewbogott: right [17:34:55] hashar: Hm, nope, renaming is broken too. Damn. [17:35:11] :-D [17:35:18] I require breakfast -- will start on this immediately once fed. [17:35:31] yeah do get a coffee + donut and everything else [17:36:21] mmmm donut [17:38:35] I think I might make paella [17:49:44] Time for dinner for me. Have a nice day! [17:49:52] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 157 processes [17:54:53] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 130 processes [17:57:47] 130... seems a lot [17:57:51] How do you get into an instance from bastion? Do you need to have a key specifically for the instance? [18:09:23] Isarra: ssh -A [18:09:36] It's using the same ssh key [18:09:56] Same as what? [18:10:04] as bastion [18:10:05] as the bastion [18:10:22] You mean the key used to log into the bastion? [18:10:28] yes [18:10:57] So the bastion needs a private key too? [18:11:23] yes, you upload it in the labsconsole web interface [18:12:22] Eeeerp. [18:12:29] login there and go to https://labsconsole.wikimedia.org/wiki/Special:NovaKey [18:14:41] Umm [18:14:50] This sounds wrong [18:15:01] I'm pretty sure if you can get into bastion your key is already in NovaKey [18:15:22] Isarra, when connecting to bastion, specify the -A argument. Then from bastion you should just be able to `ssh .pmtpa.wmflabs` [18:16:30] The public key is in the novakey thing. -A doesn't seem to change anything. [18:16:48] Isarra: so, you are on bastion? [18:16:56] Isarra: As Krenair is saying, you don't need to register a different keypair on bastion. You're going to use ssh-agent to forward your local key to bastion and onwards. [18:16:56] Aye... [18:17:02] Quit bastion. Go back to your computer. [18:17:15] Try connecting to bastion again - but this time, say -A before the hostname [18:17:28] Oh, I think I may have broken ssh-agent on my end. [18:17:42] Isarra, there are two steps. You need to set up the agent locally, and then use ssh -A [18:17:44] Would that be a problem? [18:18:11] I have a ssh config which handles it all for me, but you should learn how to do it properly :P [18:18:18] On your local machine (before you connect to bastion) do ssh-add -l [18:18:25] that will tell you if ssh-agent knows about your key or not. [18:18:53] It doesn't and it always says 'Could not open a connection to your authentication agent.' when I try to add it. [18:19:26] I take it I need to sort that out for this, then? [18:19:33] And then it should work? [18:19:40] Yep, that's the problem. [18:19:48] There's a step-by-step here: https://labsconsole.wikimedia.org/wiki/Help:Access#Using_agent_forwarding [18:19:52] (which, maybe you're doing that already) [18:21:53] Isarra I recommend everyone to create another private key on bastion - it's fast easy and secure [18:22:28] Once you are on your instances you forget the pain it was to get there... [18:22:32] :) [18:22:33] you don't need to forward anything [18:22:52] Silke_WMDE_ it was never pain for some... [18:22:58] :p [18:23:05] who followed the manuals [18:23:17] or who eventually wrote them XD [18:23:40] hehe, in the beginning I think I didn't even find them. [18:23:56] in beginning there were none :D [18:24:07] !ssh [18:24:07] https://labsconsole.wikimedia.org/wiki/Help:SSH [18:24:46] @search ssh [18:24:46] Results (Found 7): pageant, ssh, socks-proxy, git-puppet, sshkey, tunnel, -f, [18:24:53] now we have tons [18:25:27] petan: btw do we have a list of commands I can ask the wmbot? [18:25:35] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [18:26:05] wow [18:26:22] @infobot-detail !log [18:26:22] Info for !log: this key was created at 9/25/2012 6:19:38 PM by Damianz, this key was displayed 2 time(s), last time at 11/26/2012 11:01:49 PM (70.19:24:33.2352740 ago) this key is normal [18:26:32] Damianz I knew it [18:26:35] !! [18:26:35] petan needs a new hobby :P [18:26:56] wm-bot: ssh [18:26:56] Hi petan, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [18:26:59] what [18:27:07] !!log [18:27:08] petan needs a new hobby :P [18:27:15] Damianz u did that! [18:27:18] :D [18:27:19] yeah [18:27:37] Some people just randomly do [18:27:38] !! [18:27:38] petan needs a new hobby :P [18:27:39] then get spam [18:27:42] cause the bot is wimpy [18:27:51] lol [18:28:06] Blargh. [18:28:21] !! [18:28:21] petan needs a new hobby :P [18:28:31] !! [18:28:31] petan needs a new hobby :P [18:28:34] hashar are you happy for new feature i made [18:28:50] !! [18:28:50] petan needs a new hobby :P [18:28:51] works for me [18:28:52] @notify petan would be better if it just told you the line rather than telling me you where alive [18:28:57] which let me know everytime you join any channel :D [18:29:11] funny [18:29:15] Damianz you need to use it as PM [18:30:07] !damianz [18:30:07] some weirdo around here [18:30:15] here we go [18:30:53] * Damianz caws [18:31:11] Us weirdos gotta stick together [18:31:15] * Damianz looks around [18:31:23] * Silke_WMDE_ is leaving [18:31:26] ciao [18:35:23] PROBLEM Free ram is now: WARNING on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Warning: 9% free memory [19:09:16] re [19:47:19] any idea what happened to the wmflabs site and labconso;e [19:48:38] Jyothis: can you be more precise? [20:09:40] Ryan_Lane, are you around? [20:10:24] PROBLEM Free ram is now: CRITICAL on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Critical: 4% free memory [20:31:05] hashar: *.m.wikipedia.beta.wmflabs.org is now setup. The other domains will take some time [20:31:20] andrewbogott: well done! [20:32:14] http://en.m.wikipedia.beta.wmflabs.org :-] [20:32:17] not perfect yet [20:32:21] but we have progressed! [20:32:35] andrewbogott: that's awesome thank you [20:34:08] andrewbogott: ahh then the other wildcard does not resolve anymore :-D [20:34:09] en.wikipedia.beta.wmflabs.org [20:34:20] *.beta.wmflabs.org is supposed to point on deployment-squid instance [20:34:53] correct me my idiomatic is wrong, but I guess "shit has hit the fan" [20:35:44] well, let me look… there's no good reason why my changes should matter to that... [20:36:15] deployment-squid has 208.80.153.219 [20:36:32] I noticed the wildcard entry has been missing for quiet some time on https://labsconsole.wikimedia.org/wiki/Special:NovaAddress [20:36:46] maybe by creating the *.m.wikipedia.beta.wmflabs.org entry that triggered a refresh in ldap [20:36:50] which erased the wild card entry [20:38:29] So, deployment-squid is meant to be at *.beta.wmflabs.org, right? [20:39:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:40:22] PROBLEM Free ram is now: WARNING on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Warning: 17% free memory [20:40:29] hashar: ^? [20:41:18] hashar andrewbogott http://en.wikipedia.beta.wmflabs.org/ disappeared :( [20:41:32] RECOVERY Free ram is now: OK on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: OK: 21% free memory [20:43:43] andrewbogott: yeah [20:44:09] chrismcmahon: that is what I am talking about :-] The hostname en.wikipedia.beta.wmflabs.org no more point to an IP address [20:44:23] andrewbogott: should have let a note on the bug [20:44:33] hashar: yes, I read backscroll after I posted that, sorry [20:44:46] chrismcmahon: but mobile is "working" :-] http://en.m.wikipedia.beta.wmflabs.org [20:44:56] it lacks the css files which are loaded from the non mobile part [20:45:13] hashar: Do you happen to know if there was a *.wikipedia.beta.wmflabs.org entry, or if that just worked as a subcase of *.beta.wmflabs.org? [20:45:21] hashar: yes, thanks, I just told the mobile people, I saw bits aren't there yet [20:45:23] chrismcmahon: still have to get the puppet part reviewed by mark since we had to tweak the varnish conf [20:45:52] andrewbogott: I have no idea. I guess the wild card can match several subdomains [20:46:02] andrewbogott: like (en.wikipedia).beta.wmflabs.org [20:46:04] I hope so [20:46:11] ok. I'm about to break mobile again, sorry. [20:46:19] hehe :-] [20:46:30] if there is no clean solution, we might end up having to wait for ryan [20:46:39] or write the entries directly in LDAP [20:46:59] so many pings [20:48:11] yeah you tend to be popular [20:49:01] hashar: I am already writing the entries directly. There's something about the way it resolves wildcards... [20:51:20] andrewbogott: can you look at the memcache issues? basically we need to make a role, and have that role allow the specification of the memory with a variable [20:51:37] it's fine in production, this would be for labs instances [20:51:57] Ryan_Lane: What's the context for this? [20:52:03] the emails on labs-l [20:52:12] segfaults [20:52:23] memcache is configured to use more memory than the instances ahve [20:52:25] *have [20:53:10] um… 'memcached issues on piramido'? [20:53:21] yep [20:53:34] ok. I will look after I unbreak beta dns [20:53:44] cool [20:59:36] PROBLEM Free ram is now: WARNING on stackfarm-sql2.pmtpa.wmflabs 10.4.1.23 output: Warning: 17% free memory [21:02:54] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [21:03:42] Ryan_Lane, are you able to see why wikipedia.beta.wmflabs.org doesn't resolve when in other cases *.beta.wmflabs.org does? [21:04:11] are you querying the servers directly? [21:04:32] You think it's just a stale cache? [21:04:57] well, querying the servers directly would let you know ;) [21:05:21] yeah, let's see… is that -h? [21:06:12] oh, @ [21:06:45] ok, sure enough, it was cached. [21:10:42] cool [21:10:52] Ryan_Lane, when defining domains in ldap, it looks like we can't have multiple domains that start with the same thing, e.g. m.foo.beta.wmflabs.org and m.bar.beta.wmflabs.org because both of them would need to be the dn: dc=m,ou=hosts,dc=wikimedia,dc=org record [21:10:56] is that wrong? [21:13:08] I think it creates it under other entries [21:13:43] The web interface just refuses to do it. I'm trying to figure out if there's a correct thing for it to do. [21:13:57] yeah, the web interface is a little broken for this [21:15:36] I'm not sure it's just the web interface... [21:15:46] but I'm doing multiple things at once, so could be conflating issues. [21:15:55] check out other domain entries [21:16:00] I don't think they are all at the root [21:16:07] I think they exist under other entries [21:19:16] Looks like they are all at the root, but they have arbitrary record names. e.g. m.wikimediafoundation.wmflabs.org is called m-wikimediafoundation,ou=hosts,dc=wikimedia,dc=org [21:19:23] ah [21:19:48] I guess that's easy… I didn't realize the name was arbitrary. [21:19:52] * andrewbogott tries this again [21:25:42] hashar: can you fix the sudo issue on beta pleae? [21:26:03] which issue ? [21:26:45] i-00000390.pmtpa.wmflabs : Feb 5 21:26:26 : mwdeploy : 3 incorrect password attempts ; TTY=pts/2 ; PWD=/ ; USER=apache ; COMMAND=/usr/local/bin/mwscript mergeMessageFileList.php --wiki=aawiki --list-file=/home/wikipedia/common/wmf-config/extension-list [21:26:57] ahh [21:27:08] must be some manifest which is screwed [21:27:20] the mw-update-l10n script run as mwdeploy [21:27:30] and that script attempt to sudo as user mwdeploy [21:27:34] (iirc) [21:34:06] sometimes I think we're all crazy and the world should just end [21:36:38] hashar, chrismcmahon, in theory DNS is now all as it should be -- I've set up all the m.* domains, and en.wikipedia.beta.wmflabs.org is resolving properly for me on the dns server so it should trickle down to you eventually. [21:36:49] I'm pretty surprised that I'm not seeing it updated already though... [21:36:57] thanks andrewbogott [21:37:23] nothing surprises me anymore - not even users [21:39:06] andrewbogott: positive cache is 1h [21:39:24] ok, that fits. [21:39:26] Hmmm [21:39:27] Hmmm, does gerrit have a concept of owners? ie people who see CRs on their dashboard/get emails for it and auto added as reviews for specific repos? [21:39:42] I don't believe so [21:39:51] andrewbogott: great thanks! [21:39:51] at least http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page is back around [21:40:08] and http://en.m.wikipedia.beta.wmflabs.org gives out something :-] [21:40:13] Damianz: may I answer that in #mediawiki ? [21:40:31] 1sec [21:40:38] andrewbogott: that is a work for me. Thanks a ton! [21:40:42] yes [21:40:42] hashar: en.wikipedia isn't working for me yet, but I'm cheered that you can see it [21:42:05] andrewbogott: both labs name servers points en.wikipedia.beta.wmflabs.org to 208.80.153.219 which is the deployment-squid instance [21:42:16] so yeah that is fine [21:43:53] andrewbogott: you had to hack the entries directly in LDAP is that correct? [21:44:01] the mobile one don't show up on https://labsconsole.wikimedia.org/wiki/Special:NovaAddress [21:44:07] nope, there turned out to be a way to do it via the web interface [21:44:13] oh [21:44:17] but, yeah, they don't show up on reading. I'm going to try to debug that now... [21:44:28] I have marked https://bugzilla.wikimedia.org/show_bug.cgi?id=44669 :-] [21:45:30] chrismcmahon: so we have some experimental beta mobile website at http://en.m.wikipedia.beta.wmflabs.org/ thanks to andrew and maxsem :-] [21:45:44] hashar, mostly to you;) [21:45:58] bah I just copy paste ideas from other people [21:46:19] MaxSem: still have to get the varnish manifest validated but I guess we are very close to having mobile as officially in beta :-] [21:51:14] andrewbogott: thanks again! Heading bed myself [21:51:17] * hashar waves [22:55:32] PROBLEM Free ram is now: CRITICAL on spellcheckself-bot.pmtpa.wmflabs 10.4.0.246 output: Critical: 4% free memory [23:07:35] Hi andrewbogott! I think I should have invited you for review which I forgot... m) It would be amazing if you could have a look at this one: https://gerrit.wikimedia.org/r/#/c/47576/ [23:13:11] Silke_WMDE: It looks OK to me, although I'll have to take your word for it working… the specifics are a bit over my head [23:15:27] andrewbogott: Yes, I tested it. [23:15:40] good enough for me :) [23:15:42] Thanks for your time! :) [23:17:21] * Silke_WMDE is off to bed... Zzzzzz