[12:18:40] PROBLEM dpkg-check is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:21:43] petan: hello :-D I have migrated the 'beta' configuration to use operations/mediawiki-config git repo :-D [12:21:49] still have like 4 files to merge [12:21:52] nice [12:22:32] I had a discussion with Ryan Kaldari yesterday, he is willing to install some extension from the Editor Engagement program [12:23:30] RECOVERY dpkg-check is now: OK on mobile-testing i-00000271 output: All packages OK [12:35:43] ah, Ubuntu package upgrade for "facter" (puppet) in lucid. hmm.. want precise anyways though [12:36:03] hashar: that's ok [12:37:43] mutante: do you know how we can tag a Debian package for both lucid and precise ? [12:37:48] i see we have this #landscape-client (Landscape is a web-based tool for managing Ubuntu systems.) on labs instances. do you guys use that at all? [12:37:53] mutante: is that done in debian changelog ? ;) [12:38:06] mutante: no idea, I don't use it for sure [12:38:42] mutante: I guess we should drop the smaller instances or make their / bigger, 1.4GB is not enough apparently [12:38:56] hmm, i guess we should either not have it, or tell labs users about the option to use it [12:39:10] it is probably installed by default [12:39:55] mutante: I did a puppet class which clean out apt cache once per week https://gerrit.wikimedia.org/r/#/c/12151/ :-] [12:40:06] not sure if it is wanted though [12:41:14] hashar: i think you just upload it twice, once per release [12:41:28] ohh [12:41:37] but ask Faidon :) [12:41:40] I guess I will catch up with paravoid [12:41:41] hehe [12:41:44] in sync again! [12:41:47] heh [12:43:50] http://wikitech.wikimedia.org/view/Git-buildpackage#Experiments [13:08:13] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [13:16:33] PROBLEM dpkg-check is now: CRITICAL on mobile-testing i-00000271 output: Connection refused or timed out [13:28:53] PROBLEM Current Users is now: WARNING on bastion1 i-000000ba output: USERS WARNING - 6 users currently logged in [13:52:18] hey Ryan_Lane [14:14:43] PROBLEM Current Load is now: WARNING on incubator-apache i-00000211 output: WARNING - load average: 18.75, 15.30, 9.08 [14:24:53] PROBLEM Total Processes is now: WARNING on incubator-apache i-00000211 output: PROCS WARNING: 159 processes [14:27:13] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [14:32:03] RECOVERY Free ram is now: OK on mobile-testing i-00000271 output: OK: 84% free memory [14:33:53] RECOVERY Current Users is now: OK on bastion1 i-000000ba output: USERS OK - 5 users currently logged in [14:34:43] PROBLEM Current Load is now: CRITICAL on incubator-apache i-00000211 output: CRITICAL - load average: 31.21, 28.33, 21.25 [14:51:53] PROBLEM Current Users is now: WARNING on bastion1 i-000000ba output: USERS WARNING - 6 users currently logged in [15:11:53] RECOVERY Current Users is now: OK on bastion1 i-000000ba output: USERS OK - 5 users currently logged in [15:54:24] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [16:02:23] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 72 MB (5% inode=57%): [16:39:13] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [16:52:53] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [17:11:11] anybody know if changes to the puppet groups are logged anywhere? [17:11:23] what do you mean? [17:11:28] https://labsconsole.wikimedia.org/wiki/Special:NovaPuppetGroup [17:11:33] "manage puppet groups" [17:11:41] I don't think so [17:11:59] If I add (or remove) a class to one of the groups in 'all projects', it's not logged anywhere? [17:12:10] I don't think so [17:12:11] I was sure I added the labsdebrepo class to the building group last week but it was gone this morning. [17:12:31] I added it again, but if it went away for some reason, it'd be nice to know so that I can fix whatever that reason might be. [17:12:51] (alternatively, if I did it wrong, then I probably did it wrong again and that'd also be nice to know). [17:13:20] that's weird [18:32:39] !log deployment-prep Uninstalled the pecl PHP parsekit extension, manually installed php5-parsekit package instead {{bug|37076}} [18:32:40] Logged the message, Master [18:34:25] that was fast :) [18:40:46] what's the difference between the 'nova' and 'pmtpa' "Availability Zone"? I created 6 instances last night and it seems 4 of them are in the nova zone and 2 are in the pmtpa zone. [18:40:59] I see there's a pulldown menu on instance creation that lets me choose... [18:42:38] I think they're equivalent; no idea why there are two though [18:51:25] PROBLEM Current Users is now: WARNING on bastion-restricted1 i-0000019b output: USERS WARNING - 6 users currently logged in [19:10:15] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [19:14:55] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-bits i-00000264 output: DPKG CRITICAL dpkg reports broken packages [19:16:59] !log deployment-prep Removed role::cache::bits from deployment-cache-bits. Only work in production. [19:17:00] Logged the message, Master [19:17:55] RECOVERY Puppet freshness is now: OK on deployment-cache-bits i-00000264 output: puppet ran at Tue Jun 26 19:17:39 UTC 2012 [19:24:10] PROBLEM host: deployment-cache-bits is DOWN address: i-00000264 CRITICAL - Host Unreachable (i-00000264) [19:27:24] do we have a way to access an instance console? [19:34:30] RECOVERY host: deployment-cache-bits is UP address: i-00000264 PING OK - Packet loss = 0%, RTA = 2.81 ms [19:34:55] what's the best way to request a new project? for the following: http://meta.wikimedia.org/wiki/OpenMeetings.org [19:36:21] specifically, to launch a development-only MetaVidWiki instance [19:36:30] RECOVERY dpkg-check is now: OK on deployment-cache-bits i-00000264 output: All packages OK [19:39:40] !log deployment-prep deployment-feed is now out of disk space :-( [19:39:42] Logged the message, Master [19:46:45] !log deployment-prep deployment-feed removed some old kernels apt-get remove --purge linux-image-2.6.32-318-ec2 linux-image-2.6.32-342-ec2 linux-image-2.6.32-38-virtual linux-image-2.6.32-34-virtual [19:46:47] Logged the message, Master [19:48:30] RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK [19:49:17] oh [19:49:20] we have wireshark installed [19:50:57] !log deployment-prep deployment-feed removed wireshark then ran 'apt-get auto remove' , various X11 packages got removed. Now up to 262MB free. [19:50:58] Logged the message, Master [19:52:19] hashar: I think you can get console output through labsconsole. [19:52:43] maplebed: yup tried that bug got a timeout several time :-D [19:52:47] maplebed: the box is probably dead [19:52:51] (well instance) [19:54:24] I'm pretty sure that's the only path to console output in labs. [19:54:44] most probably [19:56:20] RECOVERY Current Users is now: OK on bastion-restricted1 i-0000019b output: USERS OK - 5 users currently logged in [20:13:26] RECOVERY Puppet freshness is now: OK on deployment-feed i-00000118 output: puppet ran at Tue Jun 26 20:13:13 UTC 2012 [20:13:48] !log deployment-prep Removed misc::mediawiki-logger from deployment-feed. Was replaced by some new udp2log system I can't understand. So for now, -feed is locally hacked and does not rely on puppet anymore. [20:13:49] Logged the message, Master [20:16:21] !log deployment-prep deleted deployment-syslog instance. It is of no use till we have a way to setup syslog server on labs {{bug|36748}} (syslog-ng conflict with rsyslog from base::??? puppet class) [20:16:22] Logged the message, Master [20:24:55] Change abandoned: Hashar; "Cant remember what that change was for and I don't need it. So just abandon it." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/7253 [20:52:54] Hello all! [21:52:36] PROBLEM dpkg-check is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [21:54:56] PROBLEM Total Processes is now: CRITICAL on incubator-apache i-00000211 output: PROCS CRITICAL: 202 processes [21:56:26] PROBLEM Free ram is now: WARNING on incubator-apache i-00000211 output: Warning: 18% free memory [22:21:45] PROBLEM Free ram is now: CRITICAL on incubator-apache i-00000211 output: Critical: 3% free memory [22:30:12] PROBLEM Current Users is now: CRITICAL on incubator-apache i-00000211 output: CHECK_NRPE: Socket timeout after 10 seconds. [22:31:52] PROBLEM HTTP is now: CRITICAL on incubator-apache i-00000211 output: CRITICAL - Socket timeout after 10 seconds [22:31:52] PROBLEM Disk Space is now: CRITICAL on incubator-apache i-00000211 output: CHECK_NRPE: Socket timeout after 10 seconds. [22:32:52] PROBLEM dpkg-check is now: CRITICAL on incubator-apache i-00000211 output: CHECK_NRPE: Socket timeout after 10 seconds. [22:33:02] PROBLEM SSH is now: CRITICAL on incubator-apache i-00000211 output: CRITICAL - Socket timeout after 10 seconds [22:45:17] is there anyone fulfilling requests for developer access at http://mediawiki.org/wiki/Developer_access? i recall seeing an email on wikitech-l from sumana a few days ago mentioning she'd be especially busy from then until mid-august. i've had a request there since last thursday night PDT [23:19:33] there are several people able to create developer accounts [23:20:33] I don't think sumana will be going to pay too much attention to that queue in the following weeks [23:20:50] although she may actually be! [23:21:40] on her email she states robla is her backup for all but gsoc, which would include creating dev accounts [23:21:58] personally, I'd just bug Ryan to fulfill it :P [23:22:34] Reedy or mutante may also be able to [23:23:07] Platonides: alrighty, thanks [23:23:56] i just got my extension working in mediawiki 1.19, so i think it's ready for labs [23:24:02] this seems to be the list of people: https://labsconsole.wikimedia.org/w/index.php?title=Special:ListUsers&group=accountcreators [23:24:29] I'd make an account for you, but I'm not there :) [23:24:33] awesome [23:25:10] I can do it [23:26:18] you were lucky :) [23:26:21] good night [23:27:14] thanks Reedy [23:27:35] Emw: check your email [23:28:24] got it [23:28:48] --- github.com ping statistics --- [23:28:49] 60 packets transmitted, 0 received, 100% packet loss, time 58999ms [23:29:01] Only on one labs instance [23:29:04] and i'll add you to bastion [23:29:24] marktraceur: I'm not sure all hosts are supposed to be able to connect to the outside world [23:29:43] Reedy: I've gotten things before, dependencies and whatnot [23:29:48] oh, ok then [23:29:53] is it dead to everywhere? [23:29:57] Hm [23:30:16] Apparently [23:30:25] do you know when it last worked? [23:30:35] Hm, not off the top of my head [23:30:37] I know there were some network changeabouts [23:30:45] I think today, actually [23:30:55] I would have gotten various git updates earlier today [23:31:14] From gerrit, though, so it might have been not-outside? I don't know how the network is configured [23:32:04] thanks Reedy [23:33:15] Yeah, Reedy, gerrit.wikimedia.org is working perfectly fine, google.com and github.com are totally broken [23:33:44] but that's internal... I know on our production machines, the majority don't have external access [23:34:37] Reedy: I've installed things from apt sources at ubuntu.com, which is no longer working [23:34:52] heh [23:34:52] fun [23:34:56] So I guess the last known working date was the last time I pulled from gitorious [23:35:06] mmm [23:35:36] Too late for Ryan/Faidon/Antoine.. Leslie is sick... [23:35:43] andrewbogott: do you know? [23:35:57] Reedy: catching up... [23:36:16] Last known working date: 2012-06-21 (last Thursday) at 14:40 or so [23:36:59] Reedy: Sorry, I can't tell what you're asking. Account creation? [23:37:02] I think it was before then when the changes were made [23:37:08] andrewbogott: external internet access for labs vms [23:37:36] andrewbogott: etherpad.wmflabs.org can't access the outside world [23:37:53] Hm... should work, I think. Give me a moment to look around. [23:41:01] Krinkle: That was quite an entrance [23:42:05] marktraceur: I'm taking care of some nicks that freenode removed [23:42:05] before they are abused. since by bots are still down. [23:42:20] Ah! [23:42:24] Gotcha [23:44:53] PROBLEM Current Users is now: WARNING on bastion-restricted1 i-0000019b output: USERS WARNING - 6 users currently logged in [23:45:48] Reedy: I agree that labs instances can't reach out, and that they used to be able to. [23:45:57] I'm unclear as to whether or not that's on purpose. Signs point to 'yes'. [23:46:18] In which case I can only assume there's some way to set up a security rule in order to escape... [23:46:41] mmm [23:46:54] I know production is like that, but labs being a bit more relaxed would make some sense [23:49:45] Reedy: andrewbogott: I'd be happy with an Houdini rule [23:50:14] We've got unicorns for magic [23:50:36] marktraceur, Reedy, are you both subscribed to Labs-l? [23:50:37] Reedy: This isn't magic, it's escape artistry! Hence, Houdini [23:50:44] andrewbogott: Not right now [23:51:42] andrewbogott: Yes [23:51:55] I will start a thread there, shortly. [23:54:47] andrewbogott: I'm going to head out for today, I'll read your message when I get home hopefully! [23:56:03] ok