[00:00:18] Anyone know why http://deployment.wikimedia.beta.wmflabs.org/w/index.php?title=Special:UserLogin isn't working? [00:01:06] Probably because Ryan_Lane is testing gitdeploy currently and thus beta isn't running master as normal [00:01:23] define not working would be useful though [00:01:36] not loading…I'm getting errors [00:01:41] indeed [00:01:44] that's why [00:01:47] No backend defined with the name `global-multiwrite`. [00:01:48] Backtrace: [00:01:48] #0 /srv/deployment/mediawiki/slot0/extensions/ConfirmEdit/FancyCaptcha.class.php(11): FileBackendGroup->get('global-multiwri...') [00:01:49] etc [00:01:53] MJ94: in which way is it failing? [00:02:05] hm [00:02:10] I wonder if the config is different [00:02:12] Ryan_Lane: http://puu.sh/1OdS2 [00:02:14] works for me... probably getting squid though [00:02:25] Damianz: log in is working? [00:03:00] create is failing for me [00:03:03] don't have a valid login for beta atm I don't think, test/test says invalid which is usually enough to tell if it's broken outside of cache [00:03:14] create is broken yeah [00:03:46] it move to /srv to work around gluster's hatrid of small files? [00:04:11] no [00:04:18] because that's the new location in production [00:04:21] :o [00:04:35] no more random apache/php dir?! [00:04:35] win [00:04:50] the new system is slightly more sane. only slightly, though [00:05:01] well you do run mediawiki [00:05:05] that takes out like 60% of sanity [00:05:37] * Damianz ducks from eedy [00:06:22] hmm also going to bed [00:10:43] !logs [00:10:43] logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [00:11:41] Ryan_Lane: ping [00:11:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.26, 7.35, 5.85 [00:11:49] MJ94: ? [00:11:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 6.11, 6.48, 5.64 [00:12:03] MJ94: reedy is looking at this in #wikimedia-operations, I believe [00:12:20] Ryan_Lane: is deployment only one part of labs? [00:12:31] If my connect wasn't going so slooooooow [00:12:41] beta is a project in labs [00:12:51] Damianz: heh [00:13:06] I'm working on git deploy in operations [00:13:15] since I'm doing both beta and production [00:13:26] Damianz: beta = http://deployment.wikimedia.beta.wmflabs.org/wiki/ ? [00:13:32] Standard wikimedia strategy? deploy then run [00:13:38] MJ94: That's one wiki on beta [00:13:45] Holy [00:13:50] Beta is effectivly production with a selection of data imported [00:14:03] http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page for example is another wiki [00:14:15] So there's many different labs, where beta is one lab, and beta has many wikis. [00:14:24] So basically Labs has a lot of wikis [00:14:31] not really [00:14:48] beta is a set of wikis inside of the deployment-prep project [00:14:52] Labs has projects, one project is a production clone called beta. There is other proejcts such as bots that don't have any wikis [00:15:01] Ahah! I drew a picture for this exact question: https://labsconsole.wikimedia.org/wiki/Help:Getting_Started [00:15:04] !resource deployment-prep [00:15:04] https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment-prep [00:15:12] andrewbogott: YAAAAY PICTURES [00:15:14] hah. awesome [00:15:21] andrewbogott: that's great [00:15:35] I wish project contained more than instance [00:15:39] We might get there one day [00:15:48] MH94: So, labs is made of projects, projects are made of instances. 'beta' aka 'deployment-prep' is one project. [00:16:21] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Deployment-prep < project info [00:16:49] https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/mainlabel%3D-2D/searchlabel%3Dprojects/offset%3D0 < all projects in labs [00:17:09] Ryan_Lane: Ganglia borked again :( [00:17:14] pile of crap [00:17:18] it's been borked for a while now [00:17:25] someone needs to spend some time on it to fix it properly [00:17:27] well it's more borked than it was [00:17:29] :D [00:17:32] :( [00:17:41] think it needs some TLC [00:17:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 5.50, 5.93, 5.25 [00:17:43] ah [00:17:49] thanks Ryan_Lane Damianz andrewbogott [00:17:50] since it never graphed when it wasn't erroring [00:18:45] irony that we have an ipv6 test project yet don't support ipv6 in labs [00:18:49] lawl [00:19:52] PROBLEM host: ee-lwelling.pmtpa.wmflabs is DOWN address: 10.4.0.243 CRITICAL - Host Unreachable (10.4.0.243) [00:21:06] we only use 14.5tb of storage? meh [00:23:53] RECOVERY host: ee-lwelling.pmtpa.wmflabs is UP address: 10.4.0.243 PING OK - Packet loss = 0%, RTA = 0.64 ms [00:24:23] PROBLEM Total processes is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [00:25:52] PROBLEM Current Load is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [00:25:53] PROBLEM dpkg-check is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [00:26:37] PROBLEM Current Users is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [00:27:12] PROBLEM Disk Space is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [00:28:02] PROBLEM Free ram is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [00:32:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 3.56, 4.37, 4.87 [00:35:32] PROBLEM dpkg-check is now: CRITICAL on building.pmtpa.wmflabs 10.4.0.97 output: DPKG CRITICAL dpkg reports broken packages [00:36:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.41, 4.54, 4.97 [00:37:12] RECOVERY Disk Space is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: DISK OK [00:38:02] RECOVERY Free ram is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: OK: 749% free memory [00:38:53] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 20% free memory [00:39:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [00:39:22] RECOVERY Total processes is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: PROCS OK: 99 processes [00:39:32] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 22% free memory [00:40:53] RECOVERY Current Load is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: OK - load average: 0.72, 0.92, 0.77 [00:40:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 20% free memory [00:40:53] RECOVERY dpkg-check is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: All packages OK [00:41:33] RECOVERY Current Users is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.0.243 output: USERS OK - 1 users currently logged in [00:51:44] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.45, 3.35, 4.61 [00:52:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [00:56:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 17% free memory [01:06:14] Whats the logging format? [01:08:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [01:09:42] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 179 processes [01:10:36] !log help [01:10:52] hm... [01:10:57] labs-morebots, you feeling ok? [01:10:57] I am a logbot running on i-0000015e. [01:10:57] Messages are logged to labsconsole.wikimedia.org/wiki/Server_Admin_Log. [01:10:58] To log a message, type !log . [01:14:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes [01:14:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 5.15, 5.56, 5.09 [01:19:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.16, 5.64, 5.14 [01:44:33] PROBLEM Current Users is now: WARNING on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: USERS WARNING - 6 users currently logged in [01:45:24] RECOVERY Total processes is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: PROCS OK: 150 processes [01:54:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.61, 4.42, 4.97 [02:07:47] !log deployment-prep Created geo_tags tables on all deployment-prep wikis [02:07:49] Logged the message, Master [02:09:33] RECOVERY Current Users is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: USERS OK - 5 users currently logged in [02:29:15] Hello everybody, Fox Wilson and I made shell requests for VoxelBot - https://labsconsole.wikimedia.org/wiki/Shell_Request/Vacation9 and https://labsconsole.wikimedia.org/wiki/Shell_Request/Fox_Wilson [02:29:24] If anybody could take a look, that would be great. [02:36:29] Oh also, are we autoadded to Bastion or do we need to add ourselves or does an admin need to? [02:36:49] you're auto-added to bastion, but someone needs to approve you for bots i think [02:36:59] We don't need to be added to bots [02:37:07] We will be creating our own project [02:37:42] Vacation9: why your own project? [02:38:14] Ryan_Lane: We thought it would be easier to manage since it's a multi-user effort [02:38:20] bots is as well [02:38:23] PROBLEM Total processes is now: WARNING on aggregator1.pmtpa.wmflabs 10.4.0.79 output: PROCS WARNING: 151 processes [02:38:29] Ryan_Lane: bots is fine then [02:38:33] and we're tracking dependencies and such [02:38:40] so that deploying new bots nodes is easier [02:38:44] Sounds good then [02:38:53] I understand [02:39:59] Ryan_Lane: Would you happen to be a labs admin? [02:40:08] I've done your shell requests [02:40:18] you should talk to Damianz and petan about bots [02:40:32] Alright, will do. [02:54:34] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 19% free memory [03:27:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 5.12, 5.51, 5.17 [03:32:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.57, 4.73, 4.96 [03:48:23] RECOVERY Total processes is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: PROCS OK: 150 processes [04:19:43] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [04:37:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [04:38:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [04:39:32] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 34% free memory [04:39:42] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 6% free memory [04:55:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [04:56:53] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 21% free memory [04:59:43] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [05:09:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 16% free memory [05:11:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [05:52:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 4.53, 5.31, 5.01 [05:57:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.45, 4.94, 4.95 [06:10:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 6.37, 5.91, 5.36 [06:28:55] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:30:40] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 19% free memory [06:31:25] PROBLEM Total processes is now: WARNING on aggregator1.pmtpa.wmflabs 10.4.0.79 output: PROCS WARNING: 155 processes [06:38:53] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 147 processes [06:41:54] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 32% free memory [06:46:22] RECOVERY Total processes is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: PROCS OK: 150 processes [06:50:34] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 50% free memory [07:07:24] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [07:15:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.64, 4.59, 4.94 [07:43:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 4.74, 5.28, 5.33 [07:53:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.67, 4.46, 4.82 [08:40:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [08:45:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.69, 6.25, 5.49 [09:00:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 4.38, 4.36, 4.86 [09:03:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [09:04:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [10:02:29] !log bots inserting Fox Wilson to project [10:02:31] Logged the message, Master [10:02:46] @labs-user Fox Wilson [10:02:46] That user is not a member of any project [10:02:50] aha [10:03:03] !log bastion giving access to Fox Wilson [10:03:03] Logged the message, Master [10:03:26] !log bastion Failed to add Fox Wilson to bastion. [10:03:27] Logged the message, Master [10:04:23] !log bots inserting Vacation9 to project [10:04:24] Logged the message, Master [10:05:05] @seenrx Vacation [10:05:05] petan: Last time I saw Vacation9 they were quitting the network with reason: no reason was given at 1/16/2013 3:21:49 AM (06:43:15.6284440 ago) (multiple results were found: Vacation9881) [12:22:43] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 154 processes [12:38:24] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [12:39:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 20% free memory [12:41:53] PROBLEM dpkg-check is now: CRITICAL on mobile-osm.pmtpa.wmflabs 10.4.0.226 output: DPKG CRITICAL dpkg reports broken packages [12:56:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [12:57:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [12:58:10] !log Bots Now running VoxelBot on bots-4 [12:58:10] Bots is not a valid project. [12:58:21] ? [12:58:31] !log bots Now running VoxelBot on bots-4 [12:58:33] Logged the message, Master [12:58:37] there we go [13:06:52] RECOVERY dpkg-check is now: OK on mobile-osm.pmtpa.wmflabs 10.4.0.226 output: All packages OK [13:26:54] Vacation9 do you have access to bots now? [13:27:11] I think the other guy do not yet, because he doesn't have shell [13:27:16] @labs-user Vacation9 [13:27:16] Vacation9 is member of 1 projects: Bots, [13:40:33] PROBLEM Current Users is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:40:44] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:40:53] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:42:09] oh sorry missed you Petan [13:42:14] petan: yes I do [13:42:23] Fox Wilson does as well, he does have shell [13:43:23] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:43:33] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:44:07] @labs-user fwilson [13:44:07] That user is not a member of any project [13:44:14] @labs-user Fox Wilson [13:44:14] Fox Wilson is member of 1 projects: Bots, [13:44:17] kk [13:44:43] PROBLEM SSH is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Server answer: [13:59:44] RECOVERY SSH is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [13:59:44] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 51% free memory [14:00:32] RECOVERY Current Users is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: USERS OK - 0 users currently logged in [14:00:42] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.55, 1.57, 1.39 [14:00:52] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [14:03:22] RECOVERY Total processes is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS OK: 129 processes [14:03:32] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [15:00:04] Vacation9 but without access to bastion, you can hardly login to shell [15:00:10] that's what I don't understand [15:01:20] hi, sudo asks me for password on the instance. I go through bastion1 and use ssh -A [15:01:28] i'm the admin [15:01:45] and there is a sudo all-all policy [15:02:17] yurik use console pw [15:02:28] !sudo [15:02:28] You have sudo in any project that you are a member of, excluding global projects (like bastion). Your sudo password is your labsconsole wiki password. [15:04:08] thanks petan! totally forgot :) [15:14:03] i get fatal: index-pack failed when doing git pull on /srv/mediawiki [15:14:45] on the default labs install [15:19:16] tunnel [15:19:19] !tunnel [15:19:19] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [15:24:32] PROBLEM dpkg-check is now: CRITICAL on api1.pmtpa.wmflabs 10.4.1.59 output: DPKG CRITICAL dpkg reports broken packages [15:25:09] petan: both Fox Wilson and I have bastion access [15:25:25] petan: It appears it is auto-added? [15:25:36] no idea how it works [15:25:42] We can both connect to all bots- servers, so all is well [15:25:46] it didn't work like this [15:25:48] Currently running on bots-4 [15:25:52] ok [15:26:20] you can actually use log command directly from terminal [15:29:34] RECOVERY dpkg-check is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: All packages OK [15:34:35] petan: have you seen this error before? index-pack failed ? [15:34:47] i tried git fsck --full --strict [15:34:48] that's not much of a log [15:34:56] oh git problem :D [15:35:21] it's no surprise, I get tons of them every day, usually people in #git knows [15:36:24] petan: pastebin.com/V2xuz4G7 [15:36:28] http://pastebin.com/V2xuz4G7 [15:36:37] yurik I won't help you with git, sorry :/ [15:36:47] I am really a git n00b [15:36:47] sigh :( [15:36:54] try #git [15:45:54] Vacation9 so Fox can ssh or not? [15:46:15] petan: yes it is all working fine, thank you :) [15:46:20] ok [15:46:29] don't forget to log system changes [15:46:56] what system changes should we log specifically? [15:47:22] when you install / remove packages, change users, filesystem, configuration in etc and so on [15:47:35] basically everything that could affect others [15:47:38] allright, we will. [15:47:49] are we allowed to create a usergroup for Fox and I? [15:48:09] sure [15:48:26] thanks [15:48:52] PROBLEM Current Load is now: CRITICAL on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [15:48:59] Fox is banned from freenode for some reason, freenode says he's running an open proxy :( [15:49:07] he's not [15:49:32] PROBLEM Current Users is now: CRITICAL on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [15:49:52] Oh also, does this cronjob look right? MAILTO=samuel.damashek@gmail.com [15:49:52] 0,30 * * * * /home/vacation9/VoxelBot/bot.py >/dev/null [15:50:12] PROBLEM Disk Space is now: CRITICAL on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [15:50:42] it doesn, but I don't think MAILTO will work [15:50:52] PROBLEM Free ram is now: CRITICAL on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: Connection refused by host [15:50:53] PROBLEM Current Load is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host [15:50:53] oh it won't? [15:51:02] PROBLEM Free ram is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host [15:51:05] does it have sendmail? [15:51:09] there is firewall blocking all mail going from labs in order to prevent spam [15:51:20] :( [15:51:31] I guess I'll have to output it to a log file then [15:51:32] brb [15:51:32] PROBLEM Current Users is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host [15:52:12] PROBLEM Disk Space is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host [15:52:16] How about this: 0,30 * * * * /home/vacation9/VoxelBot/bot.py >/home/vacation9/cronlog.txt [15:52:22] PROBLEM Total processes is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host [15:53:52] RECOVERY Current Load is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: OK - load average: 0.14, 0.75, 0.57 [15:54:32] RECOVERY Current Users is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: USERS OK - 0 users currently logged in [15:55:13] RECOVERY Disk Space is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: DISK OK [15:55:53] RECOVERY Free ram is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: OK: 899% free memory [15:56:03] PROBLEM dpkg-check is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host [16:00:53] RECOVERY Current Load is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: OK - load average: 0.43, 0.92, 0.65 [16:01:03] RECOVERY Free ram is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: OK: 893% free memory [16:01:03] RECOVERY dpkg-check is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: All packages OK [16:01:33] RECOVERY Current Users is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: USERS OK - 0 users currently logged in [16:02:13] RECOVERY Disk Space is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: DISK OK [16:02:23] RECOVERY Total processes is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: PROCS OK: 84 processes [16:37:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [16:41:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [16:47:49] How are you supposed to set crons on wmlabs? Our's isn't working [16:47:53] Ours* [16:47:56] crontab -e [16:48:07] so use crontab or fcrontab? [16:48:12] crontab [16:48:14] crontab has no config files set up [16:48:24] what config files should it have o.o [16:48:37] cron.allow? [16:48:56] everyone is allowed to use it by default [16:49:57] ok. We're using python, is there anything special we need to do? just do blah blah /usr/bin/python /script path? [16:50:40] if you aren't sure if it works you can always create some test record like echo "test" >> /tmp/works [16:51:21] which user are you using for cron? [16:51:34] I'm setting it from vacation9 [16:54:03] ok, let me try it [16:54:07] ls /tmp [16:54:07] ok [16:54:09] meh [16:54:15] nvm that [16:54:48] ok I inserted a record in your cron, that writes works in /tmp/test every minute and it works [16:55:05] so you should try if that command you specified there actualy does something [16:55:18] it does, that command works manually [16:55:42] in that case I recomment to you to use some startup script which produces some log so that you can find out why it didn't start [16:55:47] * recommend [16:56:00] alright then. [16:56:37] or [16:57:00] petan: thanks, the tool works now .9 [16:57:01] you could change it to /usr/bin/python /home/vacation9/VoxelBot/bot.py >> /tmp/bot.log [16:57:14] * :) [16:57:20] benestar huh? [16:57:44] petan: we talked about it yesterday evening [16:57:54] (my time) [16:57:56] aha I don't remember what was 2 hours ago so far to remember that [16:58:00] :D [16:58:05] xD [16:58:17] something with Damianzs script [16:58:33] Vacation9: I am pretty sure it produces some error message you don't see because you have no option to record the output of that job [16:58:43] by default cron redirect it to your e-mail which is blocked [16:58:57] benestar you mean that php thing? [16:59:01] yes that was fixed per logs [16:59:09] ok [16:59:11] !sal [16:59:11] https://labsconsole.wikimedia.org/wiki/Server_Admin_Log see it and you will know all you need [16:59:20] all you need xD [16:59:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [16:59:24] indeed [17:01:59] petan: we fixed it, we were not absolutely referencing a file [17:02:13] ok [17:10:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [17:38:03] could sb add me to bastion.wmflabs.org, please? [17:38:12] (User:Vogone) [17:38:37] LL-H sure, just a moment [17:39:15] LL-H: Looks like you are already. [17:39:22] oh [17:39:30] I'll have a look [17:40:02] Well, hm, you had shell rights but weren't in bastion. [17:40:12] That isn't supposed to be possible, but… should be fixed now in any case [17:40:37] thanks :) [17:41:23] yes, I had always shell access but nobody has added me to bastion ... thus I had access problems :S [17:41:51] https://labsconsole.wikimedia.org/w/index.php?title=Nova_Resource%3ABastion&diff=10265&oldid=10207 ... ty [18:03:01] Silke_WMDE: Due to a gerrit bug it's slightly hard for me to tell how your latest patchset differs from the previous one… is it just a change in the XML comments or is there more to it? [18:04:08] Hi andrewbogott! I deleted the commented in the xml file and in wikidata.pp I required git clone mediawiki before the dump is imported. [18:04:49] (or before the file is even copied to its location) [18:05:01] ok. And you're testing this as you go, I presume? [18:05:05] i do [18:05:22] Have you run a 'clean' test on a fresh instance to make sure it works without the cruft from previous tests? [18:05:49] grrrr [18:05:53] well [18:06:15] i wanted to start one but had error that did not come directly from my files [18:06:28] that was just today [18:07:30] ok… I have to go to a meeting in a moment but if that's something you want me to debug just send me an email. [18:07:40] ok, thx [18:07:48] Silke_WMDE: I'm happy to merge these changes once you're confident that they apply cleanly. [18:08:00] I'll read over them once more but I think they're fine [18:08:05] :) [18:08:40] Today I figured out variable stuff so soon there will be more (but not that much) [18:49:30] andrewbogott: so I think the beta.wmflabs.org might have a wildcard entry. [18:49:53] andrewbogott: I have tried to find out the DNS conf for beta last week but could not find where to do it in labsconsole [18:51:11] be back later, sleep time for my daugher [19:00:52] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: Connection refused by host [19:00:52] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: Connection refused by host [19:01:32] PROBLEM Current Users is now: CRITICAL on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: Connection refused by host [19:02:12] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: Connection refused by host [19:02:22] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: Connection refused by host [19:03:13] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: Connection refused by host [19:05:54] PROBLEM Current Load is now: CRITICAL on precisebuilder.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host [19:05:54] PROBLEM Free ram is now: CRITICAL on precisebuilder.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host [19:06:34] PROBLEM Current Users is now: CRITICAL on precisebuilder.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host [19:07:14] PROBLEM Disk Space is now: CRITICAL on precisebuilder.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host [19:07:24] PROBLEM Total processes is now: CRITICAL on precisebuilder.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host [19:07:54] PROBLEM dpkg-check is now: CRITICAL on precisebuilder.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host [19:10:54] RECOVERY Current Load is now: OK on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: OK - load average: 0.42, 0.99, 0.72 [19:10:54] RECOVERY Free ram is now: OK on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: OK: 898% free memory [19:11:34] RECOVERY Current Users is now: OK on testlabs-abogott-puppettest.pmtpa.wmflabs 10.4.1.50 output: USERS OK - 0 users currently logged in [19:12:14]