[00:05:02] PROBLEM - Host mw1143 is DOWN: PING CRITICAL - Packet loss = 100% [01:33:17] http://wikipedia.geo.blitzed.org/ finally stopped working [01:33:22] The end of the world :P [01:33:44] "can't determine language" PHP fatal error in /usr/local/apache/common-local/multiversion/MWMultiVersion.php line 342: [01:34:03] it is still mentioned in the apache cofnigs [01:37:07] what is that? [01:37:13] http://noc.wikimedia.org/conf/main.conf [01:37:32] ServerAlias *.wikipedia.org [01:37:33] ServerAlias wikipedia.geo.blitzed.org [01:38:05] some obscure backup domain it seems [01:38:24] http://www.gossamer-threads.com/lists/wiki/wikitech/15495 [01:41:48] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 275 seconds [01:44:35] wikipedia.geo.blitzed.org is an alias for cache.wikimedia.org. [01:44:36] cache.wikimedia.org is an alias for wikimedia-lb.wikimedia.org. [01:45:51] wow, 8 years ago [01:45:56] yeah [01:46:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [02:43:54] RECOVERY - Puppet freshness on dobson is OK: puppet ran at Tue May 29 02:43:52 UTC 2012 [03:13:00] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [03:13:00] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [03:13:00] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [03:13:00] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [03:55:00] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [04:20:57] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [06:16:52] New patchset: ArielGlenn; "fix wrong indentation so config is initialized properly" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9215 [06:17:40] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9215 [06:17:43] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9215 [06:48:19] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [06:54:19] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [06:58:13] PROBLEM - Puppet freshness on search13 is CRITICAL: Puppet has not run in the last 10 hours [06:58:13] PROBLEM - Puppet freshness on search15 is CRITICAL: Puppet has not run in the last 10 hours [07:05:16] PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours [07:05:16] PROBLEM - Puppet freshness on search16 is CRITICAL: Puppet has not run in the last 10 hours [07:05:16] PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours [07:16:13] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [07:28:54] New patchset: ArielGlenn; "option to not regenerate existing file for wiki/date" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9219 [07:30:01] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9219 [07:30:06] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9219 [07:48:10] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [07:51:01] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [07:54:38] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/9136 [07:57:19] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [07:59:16] PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours [08:19:15] New patchset: ArielGlenn; "fix nooverwrite so it works without verbose (*cough*)" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9223 [08:19:42] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9223 [08:19:44] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9223 [08:23:01] New patchset: ArielGlenn; "option to skip wikis with existing output files; cleanup verbose var" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9224 [08:23:34] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9224 [08:23:36] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/9224 [08:25:43] hey hashar [08:27:10] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [08:29:07] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [08:32:16] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [08:49:58] hello [08:50:07] database question again :) [08:50:43] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [08:50:45] i have a heartbeat table entry for i suppose s2 that seems not to be growing [08:50:57] | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | [08:50:57] +----------------------------+-----------+-----------------+-----------+-----------------------+---------------------+ [08:50:57] | 2012-05-26T10:06:16.001170 | 10623 | db13-bin.000418 | 428144675 | db52-bin.000045 | 627296604 | [08:51:06] | 2012-05-26T10:06:16.000890 | 10662 | db52-bin.000045 | 627296604 | NULL | NULL | [08:51:22] is this fine so far? [08:52:50] seems replication is still working and only slow but i have no idea why the heartbeat value is not increasing [08:53:24] ah ok the values are increasing - they are just slow [08:54:27] seems that replication is ok (as I look at http://noc.wikimedia.org/dbtree/) [09:06:27] we probably should switch to db52 as master first :D [09:07:13] *eyeroll* [10:01:31] PROBLEM - check_job_queue on neon is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_job_queue) or PHP errors - [10:02:34] PROBLEM - check_job_queue on spence is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_job_queue) or PHP errors - [10:22:23] New patchset: Hashar; "import overriding system for InitialiseSettings.php" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/9237 [10:22:29] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/9237 [10:44:24] New patchset: Bhartshorne; " giving those with sudo access acclogin access as well." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9239 [10:44:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9239 [10:45:19] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [10:47:14] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9239 [10:47:16] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9239 [12:22:25] mark: "Register by TOMORROW to participate in the Launch on 6 June." -- not sure if you want us to [12:22:32] just sayin :) [12:45:19] PROBLEM - Puppet freshness on dobson is CRITICAL: Puppet has not run in the last 10 hours [12:55:27] New review: Helder.wiki; "(no comment)" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/9237 [13:15:10] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [13:15:11] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [13:15:11] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [13:15:11] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [13:16:49] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [13:17:43] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [13:21:01] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [13:27:55] New patchset: Bhartshorne; "the group is necessary to include the accounts." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9248 [13:28:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9248 [13:28:39] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9248 [13:28:42] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9248 [13:36:14] New patchset: Bhartshorne; "Revert "the group is necessary to include the accounts." I didn't see that ldap::client::wmf-cluster is already on the host." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9249 [13:36:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9249 [13:36:42] New patchset: Bhartshorne; "Revert " giving those with sudo access acclogin access as well." I didn't see that ldap::client::wmf-cluster is already there." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9250 [13:37:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/9250 [13:37:17] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9249 [13:37:20] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9249 [13:37:26] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9250 [13:37:29] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/9250 [13:43:38] hiiii everyone! [13:43:40] i need to set up backups on stat1 [13:43:41] amanda? [13:43:43] is there a wiki page out there I should read about how to do this? [13:49:57] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.045 second response time [13:53:54] @regsearch .. [13:53:54] Results (Found 62): puppet, instance, morebots, git, bang, nagios, bot, labs-home-wm, labs-nagios-wm, labs-morebots, gerrit-wm, wiki, labs, bastion, extension, wm-bot, projects, putty, gerrit, change, wikitech, revision, monitor, alert, password, unicorn, help, bz, os-change, instancelist, instance-json, leslie's-reset, damianz's-reset, amend, credentials, queue, socks-proxy, sal, info, security, logging, ask, sudo, access, $realm, keys, $site, bug, pageant, blueprint-dns, stucked, pxe, ghsh, group, pathconflict, terminology, rt, erb, regsubst, bots, wt, gerrit-search, [13:54:03] .. [13:54:31] @infobot-link #wikimedia-labs [13:54:31] petan|wk: Unknown identifier (#wikimedia-labs [13:54:32] These channels now share the same infobot db [13:54:38] @regsearch .. [13:54:38] Results (Found 120): morebots, bang, nagios, labs-home-wm, labs-nagios-wm, labs-morebots, gerrit-wm, wiki, labs, extension, wm-bot, gerrit, change, revision, monitor, alert, password, unicorn, bz, os-change, instancelist, instance-json, leslie's-reset, damianz's-reset, amend, credentials, queue, sal, info, logging, ask, sudo, access, $realm, keys, $site, pageant, blueprint-dns, bots, stucked, rt, pxe, ghsh, group, pathconflict, terminology, etherpad, epad, nova-resource, pastebin, newgrp, osm-bug, bastion, ryanland, afk, test, initial-login, manage-projects, rights, new-labsuser, cs, new-ldapuser, projects, quilt, labs-project, openstack-manager, wikitech, load, load-all, wl, domain, docs, address, ssh, documentation, help, account, start, link, socks-proxy, requests, magic, gitweb, labsconf, console, ping, hexmode, Ryan, resource, account-questions, hyperon, deployment-prep, security, project-discuss, project-access, putty, :), instanceproject, puppet-variables, demon, linux, git, port-forwarding, pong, logs, whatIwant, broken, damianz, accountreq, puppet, report, db, nagios-fix, whitespace, instance, hashar, bot, sexytime, Thehelpfulone, bug, [13:54:39] :) [13:54:51] this is what Ryan wanted like 4 months ago :D [13:55:00] cool :) [13:55:28] so for our geodns, does anyone know where the db lives ? [13:55:39] this one? [13:55:41] on labs [13:55:47] in a file :P [13:55:50] nope, for a completely different issue :) [13:55:51] geo dns [13:55:53] aha [13:56:16] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [13:56:19] got someone in europe who was showing a few issues www.wp.org but they're getting the american ip [13:56:45] I am in europe [13:57:12] petanb@srv:~/repo/wikimedia-bot$ ping www.wp.org [13:57:12] PING www.wp.org (69.43.160.149) 56(84) bytes of data. [13:57:49] I am using google dns server [13:57:53] oh [13:57:55] wikipedia.org [13:58:01] i was being lazy and typing it wp [13:58:02] oh right [13:58:13] petanb@srv:~/repo/wikimedia-bot$ ping www.wikipedia.org [13:58:13] PING wikipedia-lb.eqiad.wikimedia.org (208.80.154.225) 56(84) bytes of data. [13:58:53] oh [13:58:55] interesting [13:59:08] what ip do you get for en.wikipedia ? [13:59:28] petanb@srv:~/repo/wikimedia-bot$ ping en.wikipedia.org [13:59:28] PING wikipedia-lb.eqiad.wikimedia.org (208.80.154.225) 56(84) bytes of data. [13:59:38] same [13:59:46] :( [14:00:00] hrm [14:00:20] so on the onehand that makes sense, on the other, i think google dns is supposed to also be passing through some ip information [14:00:36] I am pretty sure an analytic guy published the recent IP db somewhere [14:00:38] and either it's not or we're not looking at it, or something else borked with that [14:00:42] can't find out the commit / gerrit change :-( [14:01:32] LeslieCarr: do we actually use that spec? [14:01:41] I'd amazed if we did, I've never seen anyone else use it [14:02:31] hehe i don't think we do [14:02:38] in any case, aren't Google's DNS anycasted? [14:02:39] i think akamai is supposed to be using it soon ? [14:03:00] hmm [14:03:06] 8.8.8.8 is in Europe for me [14:03:15] but I get eqiad as a reply for our sites [14:03:16] yeah, though i am guessing our geoip db doesn't think it is [14:03:49] I don't think they would use the anycast IP for their outgoing queries [14:04:04] /usr/share/GeoIP [14:04:10] paravoid: 8.8.8.8 is using anycast [14:04:30] for *outgoing* queries? [14:04:31] so it is wherever the shortest BGP route sends your packet to ;-] [14:04:43] I know what anycast is and I know they're using it for 8.8.8.8 [14:05:06] what I'm asking is when they do recursion for you, if they're also using that IP to hit other servers [14:05:12] ((( and by the way, you should not use 8.8.8.8 if you care about your privacy ))) [14:05:18] I'd be very very amazed if they do [14:05:23] ohh sorry [14:05:29] they are most certainly using another address [14:05:45] since there are asymmetries in the internets, and they might lose the reply [14:05:59] hi, i worked on GeoIP stuff recently, [14:06:04] what's up? [14:06:04] well the reply might end up at another of their servers [14:06:19] and technically, they might be able to redirect it to the correct server. But that would be a bit of a mad thing to do [14:06:32] ottomata: I think LeslieCarr is trying to find out how the geoIP stuff works out [14:06:47] for DNS [14:06:59] for DNS... [14:07:12] hm, I don't know much about it for DNS I think [14:07:32] I recently changed how the .dat files were being downloaded and distributed, but afaik that is for IP address geolocating [14:08:47] anyway if some user in Europe use 8.8.8.8 as a DNS resolver AND its ISP has bad connectivity in Europe, its DNS query can totally be sent over sea to the US [14:09:23] then the Google US IP will be sent to our us and GeoIP will give back pmtpa / eqiad instead of knams :-] [14:09:27] and there is nothing we can do [14:09:48] again, I'm hitting a european 8.8.8.8 node but I'm getting eqiad as a reply for it [14:09:59] ohh [14:10:11] from it even [14:10:45] ohh [14:10:47] got same [14:11:26] my guess is that google's ips are all considered US or unknown or something [14:11:31] $ dig +short www.wikipedia.org @8.8.8.8 [14:11:31] wikipedia-lb.wikimedia.org. [14:11:33] wikipedia-lb.eqiad.wikimedia.org. [14:11:33] 208.80.154.225 [14:11:38] right [14:12:50] which is hosted somewhere behind 209.85.251.231 [14:13:06] which must be in Europe given my latency is 40ms to that node [14:13:32] damn Google does not have any reverse DNS set on its routers :-( [14:13:34] well hmm [14:13:47] X.25 was way better for that [14:15:43] oh well after looking this up more my question is moot [14:15:50] it's an indian ISP but they're going via europe [14:16:07] * hashar learned a new word: 'moot' [14:16:29] I guess from India to USA, the cheapest/ shortest route is through Europe :-((( [14:20:07] usually it's via the pacific [14:20:16] well best [14:20:19] cheapest may not be [14:20:58] !log deployment prep killing puppet on jobrunner boxes [14:21:02] ... [14:21:08] bah [14:21:10] morebot died [14:21:20] mutante: apergos can you possibly resurrect morebot? [14:21:51] again? [14:22:13] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [14:24:22] apergos: maybe it should be automatically restarted every few hours hours :-D [14:24:54] apergos: or we need to add it in some kind of a loop while(1) { morebot.sh ; sleep 60; } [14:28:59] it is [14:29:02] in a frickin loop! [14:30:03] and hereit is again [14:30:12] (I did not get over to linode so I guess th eloop got it) [14:30:18] did it [14:30:30] yer fast [14:30:33] started it in screen, then it was killed, thought that was you [14:30:57] uhm, well, as long as it lives :) not even sure now [14:34:10] or [14:34:15] you could get nagios to restart it ;-] [14:34:33] nagios has a parameter that let you execute a process to try to recover a faulty check [14:34:35] true [14:34:45] something like: /etc/init.d/apache2 restart hehe [14:34:52] I used that to reboot some routers past some uptime [14:35:43] good idea, just let's have a regular init script we can use first [14:35:49] hehe [14:36:14] upstart ftw!!! [14:36:27] supervisord ftw [14:36:35] mutante: http://upstart.ubuntu.com/cookbook/ [14:36:35] ew [14:37:08] k [14:37:18] Tivoli Workload Scheduler owns you all [14:37:26] Logged the message, Master [14:37:45] (it is just like a central cron job system) [14:40:19] hashar: lol [14:40:25] that's what I use everyday in my job [14:40:27] XD [14:40:30] ahah [14:40:45] you would earn massive amount of cash in some big french companies ;-D [14:40:54] (thought that would be a boring job haha) [14:40:59] I work for big german company :P [14:41:07] it's boring job [14:41:15] it works for me [15:00:12] ottomata: I just cherry-picked your 'generic::mysql::server' patch into labs, and could now use some advice. (I'm not totally clear on whether my question relates to your patches or not, but it seems like a good start.) Do you have a moment? [15:01:21] mutante: paravoid: I got a change to ignore /private/ directory in operations/puppet https://gerrit.wikimedia.org/r/#/q/I8b2a1864,n,z . Any of you could have a look at them please ? They are for test and production branches. [15:01:38] mutante: paravoid : useful when we have a private checkout there ;-] [15:02:30] andrewbogott_ [15:02:31] sure! [15:03:02] ottomata: Keep in mind that I know almost nothing about mysql. So probably this will turn out to be a trivial issue. [15:03:27] okey doke, what's up? [15:03:58] I've installed that class on an instance, and now I'm running the mediawiki web installer. It seems happy with all of its tests, but when it actually does the configuration it says it can't create a database. Which seems pretty basic... [15:04:06] Lemme pastebin the error page. [15:04:25] k [15:04:27] Or, actually, it's right here http://mwreview-foo.pmtpa.wmflabs/core/mw-config/index.php?page=Install (if you have a proxy set up for labs http) [15:04:59] pastebin here: http://pastebin.com/cNZ6kD14 [15:05:14] Is that just because i need to do a bit of by-hand db setup before running the mw installer? [15:05:24] hm, i don't have labs proxy set up... [15:05:32] looking... [15:06:07] Most likely I just need to tell mw the proper login/password and I'm too thick to know what that is. [15:06:14] hm, i think you need to do some by hand stuff [15:06:14] ottomata: http://dpaste.org/cKabw/ [15:06:23] the puppet stuff I set up doesn't set up any users [15:06:24] change User hashar obviously ;-] [15:06:31] so all it has is the default root no pw install [15:06:46] lemme check, I think that's what mw expects. [15:06:47] so you'd have to create whatever user (and maybe database) you need by hand [15:07:14] hashar, aye, I have an ssh proxy, but not one for http [15:07:28] ohh [15:07:36] Hm... I'm also taking for granted that the mw self-installer is stable and works. [15:08:21] ottomata: how can't you access http? Are you doing some kind of IP over DNS to bypass a paying wifi ? [15:08:50] eh? naw, I'd just need to set up an http proxy or a tunnel through bastion or whatever to get to the labs stuff [15:08:50] hashar: will that just ignore ./private or also ./foo/bla/private ? [15:08:59] ottomata, hashar: The proxy I'm talking about is this: https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy It's for accessing labs web hosts that don't have a public ip. [15:09:02] Pretty obscure. [15:09:34] mutante: hmm goo dquestion haha [15:09:48] mutante: I am pretty sure it is relative to the git repository root path [15:10:05] * hashar tries [15:10:05] hashar, I have a question for you about amanda backups, do you know much about that? [15:10:36] andrewbogott, I set up a mediawiki install recently and used the gui setup, but I think I created the database and user myself manually [15:10:36] mutante: you are right :-( [15:10:46] and entered those when mw installer asked me [15:10:56] ottomata: you got the wikitech page already? [15:11:04] yeah i found that [15:11:10] so that is easy [15:11:17] ottomata: ok. Lemme rtfm and see if I can figure that out... I'll come back and get you to hold my hand if that fails. [15:11:18] i'm editing the disklist-* files to add the host and path [15:11:21] ok cool [15:11:22] New patchset: Hashar; "git ignore /private/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6470 [15:11:27] this is /a on stat1 [15:11:32] and there is a mysql data dir in /a [15:11:37] /a/mysql [15:11:43] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/6470 [15:11:48] this mysql stuff is just temporary workspace [15:11:53] don't need to back it up [15:11:56] so I was going to exclude it [15:12:02] does that make sense? [15:12:25] I am about to commit this [15:12:25] https://gist.github.com/2828926 [15:12:48] hashar: the one on labs is already merged [15:13:00] mutante: yeah I am ending a new change [15:13:03] cool [15:13:17] mutante: https://gerrit.wikimedia.org/r/9253 (test) and patchset 2 of https://gerrit.wikimedia.org/r/6470 (prod) [15:15:49] mutante, does excluding /a/mysql like that make sense? [15:16:08] New review: Dzahn; "yea, / was the intention instead of ." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/6470 [15:16:11] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6470 [15:16:57] ottomata: Indeed, the gui installer is documented as only needing a login and password, and defaults to root w/no pass. So that's suspicious. [15:17:07] But, I will try by hand anyway... [15:17:55] ottomata: ehm, not sure if we want to keep the whole /a/ thing [15:18:10] no? [15:18:22] someone told me I should have amanda backups of stat1 [15:18:28] and ha, that's as much as I know [15:18:34] what does that mean? what should I do? [15:20:00] Ooh, I can't create a db on the commandline either. [15:20:01] mysql> create database wikidb; [15:20:01] ERROR 1006 (HY000): Can't create database 'wikidb' (errno: 13) [15:20:13] ottomata: So I must've broken your patch when I resolved conflicts :( [15:20:44] shid 9ca6e07c5fbc3e283c38b1d0ac5670116ea4ea46 [15:21:07] hmmmm [15:21:24] can you pastebin the /etc/mysql/my.cnf file [15:21:50] ottomata: f.e. ask domas about the "dammit.lt" dir in there if you get to see him [15:21:59] Yep, or you can visit the instance yourself if you want. it's called 'mwreview-foo' [15:22:59] mutante, afaik, the /a directory is just a workspace for stats related things [15:23:11] and i've been told to back it up [15:23:23] i'll ask domas, but does that mean you think I shouldn't backup /a [15:23:23] ? [15:23:24] do you already have mysql dump crons or something? [15:23:35] and then file backup the dump files [15:23:37] no, and I don't really care about /a/mysql [15:23:47] its just temp workspace [15:24:06] fabian is the only one that uses it right now, he imports some stuff, manipulates it, and then outputs files for his own use [15:24:13] ok, well, i don't know better than you who uses it [15:24:13] ottomata: paste: http://pastebin.com/6ug9qJDF [15:24:43] andrewbogott, thanks mwreview-foo says Permission denied (publickey). when I try to log in [15:24:57] this looks like a problem [15:25:00] datadir = /mnt [15:25:03] oh, hang on... [15:25:16] did you set datadir in puppet? [15:25:22] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [15:25:25] you can leave it lank [15:25:25] blank [15:25:27] for a generic install [15:25:40] include generic::mysql::server should work fine [15:25:51] and requires the least amount of manual setup [15:25:55] if you change the datadir at all [15:25:58] ok, added you to the project, I think. [15:26:03] you ahve to make sure you install a mysql database at that location [15:26:04] Catching up... [15:26:20] I do want to set the data dir, because /mnt is a different (big) volume, as opposed to the default location. [15:26:23] either copying the default /var/lib/mysql that comes with the package to the datadir, or using mysql_install_db [15:26:59] /mnt seems like a weird place for a datadir [15:27:17] Inasmuch as the class takes a datadir arg, would it be reasonable to alter the class so that it sets things up properly for an arbitrary datadir? [15:27:23] /mnt is usually an empty directory that has subdirs on which remote filesystems are mounted [15:27:29] hmmmmmm [15:27:43] Yeah... I used /mnt upon ryan_lane's advice and didn't think about it much. [15:27:44] not a bad idea, but i dunno [15:28:00] And, for labs instances, it does seem to map to an empty volume. [15:28:01] now getting Connection closed by UNKNOWN when trying to log in [15:28:31] PROBLEM - MySQL Slave Running on bellin is CRITICAL: Connection refused by host [15:28:39] hmmm [15:28:40] PROBLEM - MySQL disk space on bellin is CRITICAL: Connection refused by host [15:28:40] PROBLEM - MySQL Slave Delay on bellin is CRITICAL: Connection refused by host [15:28:40] PROBLEM - Full LVS Snapshot on bellin is CRITICAL: Connection refused by host [15:29:14] `bellin` in sounds great [15:29:16] PROBLEM - MySQL Idle Transactions on bellin is CRITICAL: Connection refused by host [15:29:28] hmm, i dunno, having puppet alter the datadir is a little scary to me [15:29:34] PROBLEM - MySQL Recent Restart on bellin is CRITICAL: Connection refused by host [15:29:34] PROBLEM - MySQL Replication Heartbeat on bellin is CRITICAL: Connection refused by host [15:30:07] maybe if it wasn't done by default [15:30:10] hashar, whatdya think about that [15:30:27] if my generic::mysql::server stuff ran mysql_install_db if the datadir was empty? [15:30:35] ottomata: The datadir is set via a role class. so it isn't modified directly in your class. [15:30:53] labsmysql.pp [15:31:15] ottomata: I am far from being a puppet guru [15:31:26] aye, but it is more a mysql question [15:31:26] I am surely not going to have any final word ;-] [15:31:40] ottomata: I am far from being a mysql guru [15:31:48] I am surely not going to have any final word ;-] [15:31:50] hahah, ok [15:31:52] ottomata: You might be able to log in now. [15:31:54] If that's still interesting. [15:31:56] (I should make that an alias ;)))))))))) ) [15:32:03] ahhhhh, man i always get hashar mixed up with binasher [15:32:13] I was here first!! ;-D [15:32:17] hashar lives in france, yes? [15:32:21] yes [15:32:24] yes! [15:32:26] and not at all a MySQL guru [15:32:54] I just know a few tricks to optimize queries and the very basic sysadmin stuff (like show process list; and settings variables) [15:33:09] When you said copy default /var/lib/mysql database... do you just mean that whole dir? [15:33:37] yeah, with cp -a [15:33:44] 'k [15:33:48] the default datadir is /var/lib/mysql [15:33:55] and [15:33:59] make sure mysql is not running when you copy it [15:34:24] so if you change the datadir, you need to make the datadir contain the same stuff as a default datadir [15:34:38] the mysql-server package sets a datadir up there [15:34:42] so you can just copy it [15:34:56] or use mysql_install_db to install a new one (the package does this when it is installed) [15:35:17] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [15:35:33] still can't get into mwreview-foo: Connection closed by UNKNOWN [15:35:46] but don't worry about it [15:38:00]