[01:02:51] uhhh can someone add me to bastion plx [01:05:53] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [01:15:53] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 20% free memory [02:24:24] !log dumps Rebooting dumps-2 [02:24:26] Logged the message, Master [03:31:29] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [03:36:29] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 15% free memory [03:37:09] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [03:41:29] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 13% free memory [03:51:29] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [03:51:29] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [03:52:09] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [03:56:29] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:01:29] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:01:29] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:02:09] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:06:29] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 94% free memory [04:27:03] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: Critical: 4% free memory [04:31:28] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [06:17:11] PROBLEM Free ram is now: WARNING on incubator-bot2 i-00000252 output: Warning: 19% free memory [09:06:43] I wish someone would have told me they couldn't get to labs :( [14:00:45] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [14:18:55] !log bots petrb: patching bot [14:18:56] Logged the message, Master [14:18:59] !log bots petrb: done [14:19:00] Logged the message, Master [14:19:09] wm-bot: ping [14:19:09] Hi petan, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [14:27:49] notpeter: that no problem - I was busy with writing up some research papers on my other work (robots) [14:28:28] notpeter: that no problem - but I'm back working on solr now and I'd like to pupetise it too. [14:28:51] notpeter: with a little help from you! [14:36:35] PROBLEM Disk Space is now: CRITICAL on e3 i-00000291 output: Connection refused by host [14:36:35] PROBLEM Current Load is now: CRITICAL on e3 i-00000291 output: Connection refused by host [14:36:35] PROBLEM Current Users is now: CRITICAL on e3 i-00000291 output: Connection refused by host [14:36:35] PROBLEM Total Processes is now: CRITICAL on e3 i-00000291 output: Connection refused by host [14:36:40] PROBLEM Free ram is now: CRITICAL on e3 i-00000291 output: Connection refused by host [14:36:45] PROBLEM dpkg-check is now: CRITICAL on e3 i-00000291 output: Connection refused by host [14:39:03] how can I reset my gerrit pass ? [14:39:38] OrenBo: on wiki [14:39:44] labs [14:39:47] Reset your labsconsole pass. [14:39:47] OrenBo: PasswordReset [14:39:56] or ResetPw idk [14:40:19] !password [14:40:19] gfgjoagaewhgAW#YAU_#Y$*U*U^*^%Q#Tqyhe [14:40:24] hmm [14:40:32] ;-) [14:40:33] wondering what is this password [14:40:44] indeed [14:40:46] https://labsconsole.wikimedia.org/wiki/Special:PasswordReset < [14:40:47] maybe that is your pw? :D [14:41:24] it says Passwords cannot be changed [14:41:30] !Ryan [14:41:30] man of all answers ever (but there are others :)) [14:43:56] petan: ? [14:44:08] you can't go directly to the form [14:44:16] you need to go to it via the login form [14:44:21] It's a session/symantic wiki bug thing [14:44:24] Ryan loves bugs [14:44:25] otherwise it doesn't set the domain [14:44:44] well, if the stupid password field had a domain drop down, it wouldn't be an issue [14:44:57] * Damianz points you towards a core dev [14:45:05] * petan hides [14:46:28] ok great I'm in again [14:47:20] Ryan_Lane: BTW that another cool "feature" of labs [14:47:45] ? [14:48:01] Damianz: they aren't going to bother to fix it [14:48:29] :( [14:48:43] OrenBo: what's a great feature? [14:48:59] if you're making a sarcastic joke, I welcome patches ;) [15:01:41] RECOVERY Disk Space is now: OK on e3 i-00000291 output: DISK OK [15:01:41] RECOVERY Current Load is now: OK on e3 i-00000291 output: OK - load average: 0.44, 0.30, 0.25 [15:01:41] RECOVERY Current Users is now: OK on e3 i-00000291 output: USERS OK - 0 users currently logged in [15:01:41] RECOVERY Total Processes is now: OK on e3 i-00000291 output: PROCS OK: 86 processes [15:01:46] RECOVERY Free ram is now: OK on e3 i-00000291 output: OK: 92% free memory [15:01:46] RECOVERY dpkg-check is now: OK on e3 i-00000291 output: All packages OK [15:19:39] New review: Dzahn; "thanks for fixing cronspam" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11753 [15:19:39] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/11753 [15:22:31] suhasmonk: hi [15:22:40] suhasmonk: you still have access to labs, right? [15:22:52] Ryan_Lane, hey! Yeah. I do [15:22:59] I'll need to update your wiki, to bring it up to the same revision that I'm working on [15:23:27] yes I remember now what I needed hashar [15:23:31] feed in labs [15:23:33] Ryan_Lane, okay. have you started working on openstack api? [15:23:37] nope [15:23:44] PROBLEM Current Load is now: CRITICAL on translation-memory-2 i-000002d9 output: Connection refused by host [15:24:03] petan: I am listening [15:24:18] we need to make that irc feed accept udp data from labs [15:24:24] PROBLEM Current Users is now: CRITICAL on translation-memory-2 i-000002d9 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:24:28] for some reason it reject it, someone with shell on prod needs to check it [15:25:04] PROBLEM Disk Space is now: CRITICAL on translation-memory-2 i-000002d9 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:25:05] I am pretty sure we have explicitly disabled UDP from labs to production [15:25:15] to avoid "beta" cluttering the production IRC channels [15:25:39] no, we didn't [15:25:44] PROBLEM Free ram is now: CRITICAL on translation-memory-2 i-000002d9 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:26:00] MY WORLD IS FALLING APART [15:26:03] we just keep disabling the setting in beta [15:26:09] yeah cause of me [15:26:17] I need a cleaner way to disable it [15:26:43] Ryan_Lane, okay. so first I'll just do the basic wrapper and push it. you can review it and let me know if its the right approach. is that okay? [15:26:51] sure [15:26:54] PROBLEM Total Processes is now: CRITICAL on translation-memory-2 i-000002d9 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:27:34] PROBLEM dpkg-check is now: CRITICAL on translation-memory-2 i-000002d9 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:28:20] hashar: so is it disabled somewhere? [15:28:25] if so, can you enable it [15:29:21] I have no idea how that part of the cluster work :-( [15:29:24] so unlikely [15:29:29] meh [15:31:48] hashar, ffmpeg2theora is not included in or by ffmpeg? [15:32:06] I remember deciding that it was last time I looked at this... but today I can not find confirmation of this. [15:33:55] I have no idea [15:34:07] just cherry picked a change to sync the production / test branches [15:34:47] Ryan_Lane: do you know someone who understand it? [15:34:53] hashar: Oh, and it's /my/ change too, isn't it? [15:35:01] I have no clue how it works [15:35:06] andrewbogott: it is is indeed :-] [15:35:21] Guess I'd better approve it then. [15:35:24] andrewbogott: git checkout production && git cherry-pick 3c6a598 && git-review [15:36:12] you are probably there because of the mail notification which sent to you by gerrit because you are that patch author :-] [15:36:34] * andrewbogott nods [15:37:18] !log deployment-prep running apt-get upgrade on apache30 and apache31 [15:37:19] Logged the message, Master [15:38:01] which is most probably going to kill both instances [15:38:02] Ryan_Lane: is there a place I could find someone who knows [15:38:15] mark would know [15:42:00] Ryan_Lane: can you create importer group on labs and give it to me :D [15:42:17] so that we can work on that wikitech merge [15:43:43] I think it exists [15:43:45] lemme add you to it [15:44:07] done [15:45:39] wm-bot: ping [15:45:39] Hi petan, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [16:18:53] I am off for now, see you later tonight [17:02:38] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 17% free memory [17:05:26] Ryan_Lane: Can you add me to the Etherpad project on Labs? [17:05:32] sure [17:05:52] Thankee [17:07:22] done. may take a min for your key to be added [17:07:28] Awesome [17:08:35] 06/18/2012 - 17:08:35 - User marktraceur may have been modified in LDAP or locally, updating key in project(s): etherpad [17:10:39] 06/18/2012 - 17:10:39 - Creating a home directory for kwisatz at /export/keys/kwisatz [17:11:39] 06/18/2012 - 17:11:39 - Updating keys for kwisatz at /export/keys/kwisatz [17:25:27] Helpful message of "failed to allocate IP address", is there some step I missed? [17:26:48] hi [17:31:30] 06/18/2012 - 17:31:30 - Created a home directory for kangaroopower in project(s): bastion [17:31:35] ^^ Kangaroopower [17:31:42] yeah I saw [17:31:43] thanks [17:31:49] !log bastion added Kangaroopower to the project [17:31:50] Logged the message, Master [17:32:35] 06/18/2012 - 17:32:34 - User kangaroopower may have been modified in LDAP or locally, updating key in project(s): bastion [17:33:44] PROBLEM Current Load is now: CRITICAL on etherpad-lite-testing i-000002da output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:34:24] PROBLEM Current Users is now: CRITICAL on etherpad-lite-testing i-000002da output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:35:04] PROBLEM Disk Space is now: CRITICAL on etherpad-lite-testing i-000002da output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:35:44] PROBLEM Free ram is now: CRITICAL on etherpad-lite-testing i-000002da output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:36:41] Just curious, but what does https://labsconsole.wikimedia.org/wiki/Help:Terminology mean when it calls an instance a virtual machine... [17:36:54] PROBLEM Total Processes is now: CRITICAL on etherpad-lite-testing i-000002da output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:37:34] PROBLEM dpkg-check is now: CRITICAL on etherpad-lite-testing i-000002da output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:38:48] <^demon|zzz> Kangaroopower: It means you're not actually working on an individual server somewhere when you make your projects--that is shared. However, it acts like an independent system with its own operating system and such separate from the other instances. So "Virtual Machine" is another way of saving "Virtual Server" [17:39:52] so like a server that doesnt exist in reality, but does in the cloud"] [17:41:04] <^demon|zzz> Yep, that's right. It's not a physical server that you could go take off a rack somewhere. [17:42:04] cool, thanks [17:42:49] <^demon|zzz> You're welcome. [17:47:44] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 5% free memory [17:55:00] Ryan_Lane: ping [17:57:54] preilly: packet refused [17:58:21] Ryan_Lane: ha ha [18:00:03] 06/18/2012 - 18:00:03 - User andrew may have been modified in LDAP or locally, updating key in project(s): testlabs,gluster,openstack,bastion,globaleducation,deployment-prep,mwreview [18:00:10] 06/18/2012 - 18:00:10 - Updating keys for andrew at /export/keys/andrew [18:10:24] PROBLEM Free ram is now: WARNING on aggregator-test1 i-000002bf output: Warning: 19% free memory [18:12:44] RECOVERY Free ram is now: OK on bots-3 i-000000e5 output: OK: 59% free memory [18:31:02] "Failed to allocate new public IP address." <- error, or because I need extra permissions? [18:35:18] might need to be a netadmin [18:35:19] OR [18:35:35] maybe some admin need to grant your project a spool of public addresses :/ [18:35:46] I know a few months ago I had to explicitly ask an address [18:36:10] * marktraceur already is a netadmin on the project [18:36:19] My guess is the latter, then [18:37:14] marktraceur: why do you need a public IP for this? [18:37:23] public ips are quotad [18:38:54] Ryan_Lane: It would be helpful to access the etherpad instance with a web browser, though if there's a preferred method, I can do that [18:39:03] use a socks proxy [18:39:04] <^demon|zzz> SOCKS :) [18:39:06] !socks_proxy [18:39:10] !socks [18:39:10] ssh @bastion.wmflabs.org -D ; # [18:39:24] you can use foxyproxy with rulesets too [18:39:31] Got it, thanks [18:39:34] yw [18:47:45] Hm, maybe not [18:49:10] Ryan_Lane: Should http://.pmtpa.wmflabs be accessible, then, after setting up the proxy and the foxyproxy rules? [18:49:28] only if you added it to a security group that has web access ;) [18:50:04] Should be [18:58:24] It's in the web security group with port 80 enabled.... [19:07:57] Well, this is awesome, the proxy thing isn't happening [19:29:36] 06/18/2012 - 19:29:36 - Updating keys for shantanoo at /export/keys/shantanoo [19:42:04] PROBLEM Free ram is now: WARNING on ganglia-test2 i-00000250 output: Warning: 19% free memory [20:15:04] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 4.75, 12.43, 8.13 [20:25:04] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 0.27, 1.90, 4.40 [20:27:14] PROBLEM Free ram is now: WARNING on ve-nodejs i-00000245 output: Warning: 11% free memory [20:37:14] PROBLEM Free ram is now: CRITICAL on ve-nodejs i-00000245 output: Critical: 3% free memory [20:42:14] PROBLEM Free ram is now: WARNING on ve-nodejs i-00000245 output: Warning: 8% free memory [21:43:19] Is someone around that could set up a labs project for Continuous integration? [21:43:26] https://bugzilla.wikimedia.org/show_bug.cgi?id=37706 [21:45:31] 06/18/2012 - 21:45:31 - Created a home directory for krinkle in project(s): jenkins [21:46:30] 06/18/2012 - 21:46:30 - User krinkle may have been modified in LDAP or locally, updating key in project(s): jenkins [21:58:22] http://ganglia.wmflabs.org/ down ? [22:04:20] !log jenkins Cleaning up project to re-use for prototyping continuous integration (TestSwarm+BrowserStack, from Jenkins) [22:04:21] Logged the message, Master [22:06:29] PROBLEM host: jenkins2 is DOWN address: i-00000102 check_ping: Invalid hostname/address - i-00000102 [22:33:45] PROBLEM Current Load is now: CRITICAL on integration-apache1 i-000002dc output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:34:25] PROBLEM Current Users is now: CRITICAL on integration-apache1 i-000002dc output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:35:05] PROBLEM Disk Space is now: CRITICAL on integration-apache1 i-000002dc output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:35:45] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002dc output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:36:15] PROBLEM HTTP is now: CRITICAL on integration-apache1 i-000002dc output: Connection refused [22:37:35] PROBLEM Total Processes is now: CRITICAL on integration-apache1 i-000002dc output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:38:15] PROBLEM dpkg-check is now: CRITICAL on integration-apache1 i-000002dc output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:40:31] 06/18/2012 - 22:40:31 - User kangaroopower may have been modified in LDAP or locally, updating key in project(s): bastion [22:40:38] 06/18/2012 - 22:40:38 - Updating keys for kangaroopower at /export/keys/kangaroopower [22:54:31] 06/18/2012 - 22:54:31 - Creating a project directory for integration [22:54:32] 06/18/2012 - 22:54:31 - Created a home directory for hashar in project(s): integration [22:55:32] 06/18/2012 - 22:55:32 - User krinkle may have been modified in LDAP or locally, updating key in project(s): integration