[00:00:36] it is out of erik's budget [00:00:58] ahh, then my comment is immaterial [00:01:05] and spaces have been allocated mostly [00:01:33] except for speakers (and only a handful left) [00:02:24] !log profiling collector was pegged at 100% cpu and graphs were turned to swiss cheese due to a bad stats call in 1.20, now fixed [00:02:26] Logged the message, Master [00:05:12] aude: I'll just wait on the waiting list, it's no big deal [00:06:09] if my talk naturally gets added, I'll go, otherwise I'll pass my travel slot to someone else in tech who hasn't gone to wikimania [00:07:48] I was just kind of surprised that the community doesn't want to hear about how bots and tools will work in this system [00:08:08] Ryan_Lane: it has to work to get you in the program :) [00:10:19] Ryan_Lane: i think i see a spot where someone said they can't come [00:57:23] PROBLEM - mysqld processes on db58 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [01:40:53] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [01:43:17] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 254 seconds [01:46:09] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 5 seconds [03:18:12] RECOVERY - mysqld processes on db58 is OK: PROCS OK: 1 process with command name mysqld [03:21:29] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 8343 seconds [03:28:50] PROBLEM - MySQL Slave Delay on db58 is CRITICAL: CRIT replication delay 7418 seconds [03:37:05] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay 0 seconds [03:37:14] RECOVERY - MySQL Slave Delay on db58 is OK: OK replication delay 0 seconds [04:32:32] PROBLEM - Host lvs6 is DOWN: PING CRITICAL - Packet loss = 100% [04:34:20] PROBLEM - BGP status on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, sessions up: 7, down: 1, shutdown: 0BRPeering with AS64600 not established - BR [04:37:30] New patchset: Jeremyb; "simplify wrapper" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5778 [04:37:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5778 [05:23:44] hm [05:23:46] anyone alive? [05:23:49] * jeremyb [05:24:00] well, anyone with access :) [05:24:07] * jeremyb not [05:28:00] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [05:28:20] paravoid: you've seen lvs6? ^^ [05:29:28] that's what I was trying to do [05:35:03] RECOVERY - Host lvs6 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [05:35:12] RECOVERY - BGP status on cr1-sdtpa is OK: OK: host 208.80.152.196, sessions up: 8, down: 0, shutdown: 0 [05:35:40] !log powercycled lvs6, was dead and not responding to serial [05:35:43] Logged the message, Master [05:35:57] so, what's the answer? [05:37:06] what's the question? :) [05:37:26] what changed in your knowledge of lvs6? [05:37:45] or was it just unfamiliar and you figured it out but slower than if someone had been around? [05:38:25] newer dracs need "console com2" instead of "connect com2" [05:38:40] how friendly [05:38:43] I was typing "connect com2" and getting a cryptic message back [09:01:00] mutante: good luck with all the boxes :-]] [09:01:21] statistically there must be one with a screwed DIMM [09:04:35] hashar: arr, thanks. actually, fail to connect to the very first one [09:04:45] told you :-D [09:05:04] try to get the other one, cause that first one might be the screwed one [09:05:34] * hashar hears a facepalm noise in The Netherlands [09:05:51] yea, but start with 1002 instead of 1001 in the naming scheme [09:47:33] New patchset: Nikerabbit; "Cron entries for TranslationNotifications" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5783 [09:47:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5783 [10:45:00] PROBLEM - check_all_memcacheds on spence is CRITICAL: MEMCACHED CRITICAL - Could not connect: 10.0.11.27:11000 (timeout) 10.0.11.32:11000 (timeout) 10.0.8.23:11000 (timeout) 10.0.8.39:11000 (timeout) [10:47:42] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [11:02:45] New review: Siebrand; "(no comment)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/5783 [11:37:41] PROBLEM - check_all_memcacheds on spence is CRITICAL: MEMCACHED CRITICAL - Could not connect: 10.0.8.23:11000 (timeout) 10.0.8.39:11000 (timeout) [11:39:11] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [11:39:48] !log running authdns-update to add analysis mgmt names [11:39:50] Logged the message, Master [11:42:20] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [12:07:37] PROBLEM - check_all_memcacheds on spence is CRITICAL: MEMCACHED CRITICAL - Could not connect: 10.0.8.39:11000 (timeout) [12:11:58] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [12:16:19] PROBLEM - check_all_memcacheds on spence is CRITICAL: MEMCACHED CRITICAL - Could not connect: 10.0.8.23:11000 (timeout) [12:19:10] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [12:29:46] !log Sending US, Brazil, Indian traffic to upload.eqiad [12:29:49] Logged the message, Master [12:32:09] New patchset: Mark Bergsma; "Silence cron spam" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5791 [12:32:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5791 [12:32:34] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/5791 [12:32:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5791 [12:33:04] hi hashar, could you maybe have a quick look at: https://gerrit.wikimedia.org/r/5717 it is about setting up stat1 for erik zachte [12:33:50] diederik: can't today sorry [12:34:36] mutante: if you can take the time to merge & restart mysql yeah please do it . Should be all about merging both changes, running puppet, crossing fingers and restarting mysql [12:34:43] ok, i'll shop around some more :) [12:35:41] hashar: ok, doing that now, cause installing these servers will take more time. i know one is already live anyways and the other has asher review [12:35:48] diederik: you will want an op to review it then merge / deploy it :-] [12:35:58] diederik: sorry, already too many stuff to track :-] [12:36:32] mutante: so you have 15% of servers installed :-] [12:36:37] diederik: you already have that:) [12:37:00] diederik: oh, no, i see, you added the right R packages..ok [12:37:46] hashar: i have preparational work like mgmt DNS entries, updated racktables, document the mgmt CLI commands .. .:P [12:41:55] New review: Dzahn; "yea, this is puppetizing a life hack which is good and should not actually change stuff and" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4395 [12:41:58] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4395 [12:44:35] New review: Dzahn; "more contint db config. has Asher review and Facebook-only config line has been removed" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4400 [12:44:38] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4400 [12:46:27] \o/ [12:46:53] hashar: but still needs a fix :p [12:47:05] error in the erb template [12:47:09] argHGH [12:47:15] !erb [12:47:15] to check the syntax of a puppet erb template: erb -x -T '-' mytemplate.erb | ruby -c [12:47:16] :) [12:47:36] mysql/log_slow_queries.cnf.erb:8: syntax error, unexpected ';', expecting ')' [12:47:55] mysql/log_slow_queries.cnf.erb:9: syntax error, unexpected ')' [12:48:39] whoever invented the ; as a line terminator back in the 1970's deserve a blame stick [12:49:32] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/5717 [12:49:35] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5717 [12:50:28] ah, thanks Mark, i was going to do that next [12:50:33] diederik: ^ [12:50:45] thanks guys! [12:56:39] !g Icda77ab48e67624ceabf2d9b7b3b259d9d84aa53 [12:56:39] https://gerrit.wikimedia.org/r/Icda77ab48e67624ceabf2d9b7b3b259d9d84aa53 [12:56:47] OH MY GOD [12:56:53] that does not work :-( [12:58:31] New patchset: Hashar; "log_slow_query mysql template was invalid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5793 [12:58:39] mutante: https://gerrit.wikimedia.org/r/5793 should fix the erg issue [12:58:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5793 [12:58:55] New review: Hashar; "ERB syntax errors corrected with https://gerrit.wikimedia.org/r/5793" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4400 [12:59:57] New review: Dzahn; "sure, syntax error fix" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/5793 [13:00:00] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5793 [13:01:21] hashar: much better:) applying configuration... [13:01:58] Contint::Test::Testswarm/File[/etc/mysql/conf.d/log_slow_queries.cnf]: Scheduling refresh of Service[mysql] [13:02:09] ahah [13:02:23] sounds like puppet is able to break mysql on its own [13:02:25] \o/ [13:03:50] so yeah, what is it waiting for right now [13:05:23] sure about the "subscribe" to the 2 files..? [13:07:33] want me to stop puppet and start mysql manually? how long is your current maintenance time ?;) [13:08:07] as short as you can :-] [13:08:16] puppet is sitting there, not stopping but not finishing either [13:08:40] I would ^C puppet and try restarting mysqld manually [13:08:46] seems down for now : https://integration.mediawiki.org/testswarm/ [13:11:24] ok, better. stopped mysql, puppet run finished [13:11:57] start: Job is already running: mysql [13:13:05] I am connecting to mysql [13:13:13] where's the PID file [13:13:16] looking [13:13:40] /var/run/mysqld empty :( [13:14:40] $ status mysql [13:14:40] mysql respawn/post-start, (post-start) process 24369 [13:14:47] service mysql restart "Since the script you are attempting to invoke has been converted ..Upstart job, you may also use the start(8) utility," [13:16:37] Misc::Contint::Test::Testswarm/Service[mysql]/ensure: ensure changed 'stopped' to 'running' [13:18:05] maybe /var/log/daemon.log has some clues ? :/ [13:18:35] gallium init: mysql post-start process (25287) terminated with status 1 [13:18:45] gallium init: mysql main process (25398) terminated with status 2 [13:19:02] init: mysql main process ended, respawning [13:21:10] 120425 13:20:13 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it. [13:21:14] hmm [13:21:17] maybe cause I am not root [13:21:23] mysqld: Table 'mysql.plugin' doesn't exist [13:21:34] how about this? leave the service { "mysql" in there but remove the subscribe to the files for now and it should be back to before, right? mysql was just started as a regular service before and no packages were changed [13:22:07] I think there is another issue [13:22:11] mysqld --help --verbose > /dev/null [13:22:16] let me paste the result of that [13:22:22] 120425 13:21:40 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it. [13:22:33] so maybe mysql got magically upgraded at some point ? :( [13:23:40] from dpkg.log 2012-04-05 13:34:23 upgrade mysql-server 5.1.41-3ubuntu12.10 5.1.61-0ubuntu0.10.04.1 [13:23:48] might not have been restarted [13:23:53] but unrelated to the puppet change.all you added was to ensure the service is running [13:24:00] yup [13:24:15] but maybe mysql was not fully upgraded [13:24:41] hence he was running in a state which would not let it restart [13:24:45] so it would have broken at next restart ..and we just triggered it, yep [13:25:06] that is what I suspect [13:26:10] how about dist-upgrading to new mysql versions we are being offered [13:26:28] you also had -ubuntu before , right [13:26:51] thats what made you remove the facebook-only config line [13:27:48] !log running apt-get upgrade on gallium [13:27:49] yeaht that is the stock one [13:27:50] Logged the message, Master [13:28:44] Setting up mysql-server-core-5.1 (5.1.62-0ubuntu0.10.04.1) ... etc... [13:28:56] anyone here can make changes to meta css file ? [13:29:09] if not any suggestions how to do that? [13:29:39] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/5768 [13:29:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5768 [13:30:24] hashar: sigh, it also kind of hangs at setting it up now :p [13:31:33] ahah [13:31:37] we are screwed :-( [13:31:45] maybe cause there is lot of rows? [13:33:18] so indeed the script is at 'start mysql' [13:33:19] :( [13:33:20] /bin/bash -e /var/lib/dpkg/info/mysql-server-5.1.postinst configure [13:34:51] killed it, dpkg was done otherwise.. at least that doesnt look broken now [13:35:49] !gallium - dpkg-reconfigure mysql-server-5.1 [13:36:50] !log gallium - dpkg-reconfigure mysql-server-5.1, mysql does not start right [13:36:53] Logged the message, Master [13:40:17] hashar: mysql> ! [13:40:51] Krinkle: there :) [13:41:01] alrighty [13:41:08] mutante: just in time!! [13:41:18] I mean before Timo could even complain about https://integration.mediawiki.org/testswarm/ being dead hehe [13:41:28] almost ;-) [13:41:39] !log gallium/testswarm - back up after mysql upgrade and issue starting the service [13:41:40] but really, no problem. the swarm clients are long-running in the browsers [13:41:41] Logged the message, Master [13:41:56] they use frames and ajax for everything, the clients won't die or hang [13:42:08] they'll just keep trying every 30 seconds until it works again and then continue as if nothing happened [13:42:10] mutante: can you check if it logs any slow queries ? [13:42:26] should be in /var/log/mysql something [13:42:39] all my swarm clients are still connected and back in the swarm now [13:43:25] hashar: let me move your config files back, i just removed them to make sure they did not cause anything..one more restart then [13:43:53] actually, let puppet do it and see if nothing happens to the service there either [13:45:23] Contint::Test::Testswarm/File[/etc/mysql/conf.d/log_slow_queries.cnf]/ensure: defined content Scheduling refresh of Service[mysql] .... [13:45:30] * hashar crosses fingers [13:45:46] it takes so long again... [13:46:54] nope, not looking good ...:( [13:49:55] so that must be one of the changes ? [13:50:24] ohh [13:50:35] log_slow_queries.cnf has a line showing 'false' [13:50:43] must be the stupid template trick [13:52:27] !log gallium stopped puppet, moved log_slow_queries config, re-setting up mysql again [13:52:29] Logged the message, Master [13:54:58] !erb [13:54:58] to check the syntax of a puppet erb template: erb -x -T '-' mytemplate.erb | ruby -c [13:55:00] hashar: its the innodb buffer log size thing [13:55:19] back up again [13:55:35] :-( [13:55:38] buffer pool size i meant [13:55:52] the erg template must be wrong somehow [13:55:59] err ERB template must be wrong somehow [13:56:16] I am fixing the log_slow_queries.cnf.erb one [13:57:37] nnoDB: WARNING: over 67 percent of the buffer pool is occupied by lock heaps or the adaptive hash index [13:57:51] well that sounds like the reason you want to change its size [13:58:03] New patchset: Ottomata; "statistics.pp - Ah, need libxt-dev in order for R to build and install Cairo R library." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5795 [13:58:06] brb, puppet wont break it for now [13:58:09] stopped the agent [13:58:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5795 [13:58:37] New patchset: Hashar; "gallium mysql templates were wrong again" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5796 [13:58:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/5796 [13:58:57] mutante: taking a coffee. https://gerrit.wikimedia.org/r/5796 should fix the template [13:59:18] need to find out how the files will be generated though [13:59:29] I probably should have tested all of that in a labs first :-( [14:00:31] hashar: same here, and actually still need to install servers, well, it is up now and you got newer mysql packages [14:00:52] that is some progress! [14:01:02] heh:) [14:01:09] I will play with labs and polish it up [14:01:11] thanks Daniel! [14:01:30] np Antoine [14:03:16] !log gallium - don't start puppet unless the erb template fix for mysql has been merged [14:03:19] Logged the message, Master [14:29:55] hioooo [14:30:03] looking into another RT ticket [14:30:11] having trouble with some NFS mounts [14:30:17] what are these two IPs? [14:30:35] 208.80.152.185, 10.0.5.8 [14:38:35] hi domas, to continue the conversation about webstatscollector, (and forgive my lack of knowledge of berkekely-db) if the db is basically all in memory, do you still need to set the DB_CREATE and DB_TRUNCATE flags when you open the handle? [14:55:50] ottomata, 208.80.152.185 seems to be dataset2 [14:58:18] thanks Platonides [15:08:12] /topic [15:12:39] mark: if you have a moment, I am going to run the mgmt connections for row c, I put the port assignments in https://rt.wikimedia.org/Ticket/Display.html?id=2859 [15:18:52] damn it, my headphones just got pulled out of my ears by catching on something, and the silicone earbud popped off and disappeared [15:18:54] ;_; [15:19:03] I hate it when that happens [15:20:04] today just went to shit [15:20:10] all cuz i have no tunes [15:20:42] You should get some cans and look like a tool but rock out to the bass [15:21:07] i prolly should, i spend so much time in the dc anyhow [15:21:16] but i hate stuff ON my ears, they get warm [15:21:42] now i get to wear a single earbud and try to fashion a temp earbud collar out of earplug material ;] [15:22:06] when does the DC meetup group get to come visit? ;) [15:22:20] I use to wear ear defenders over earbuds when spending lots of time in the dc, mainly so I could hear my phone lol [15:22:23] or wikimania field trip ;) [15:23:11] i lost mine too [15:26:45] aude: uhhhhhh [15:26:53] i guess i need to do something about that [15:27:02] im going to say i will look into it, and promptly forget again ;] [15:27:20] :) [15:27:45] i thought i emailed our eq rep about this, cuz there is a 'pbx tour guide' checkbox in the user mgmt [15:27:51] lemme see if i can find the email and resend [15:27:57] it sounds boring yet some people would think it's interesting [15:28:04] RobH: cool :) [15:28:07] New review: Hashar; "Someone needs to check that the ERB templates actually generate a valid MySQL configuration. That ca..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/5796 [15:29:00] !log hashar: gallium: MySQL had issues most probably because of the mysql configuration snippets. https://gerrit.wikimedia.org/r/5796 might solve that. [15:29:04] Logged the message, Master [15:29:04] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [15:29:49] New review: Hashar; "I attempted testing them in labs but since I have no merge rights in the test branch, I can't get th..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/5796 [15:29:55] maaaaark mark mark mark [15:30:02] maybe you have info for me about this [15:30:03] yes? [15:30:10] i'm working on this ticket [15:30:10] http://rt.wikimedia.org/Ticket/Display.html?id=2162 [15:30:19] trying to mount two machines on the new stat1 server [15:30:36] aude: resent the email to our EQ rep, I know I wanted to get something setup for the board, organizers, etc. [15:30:47] but there is no way it will ever be an open 'everyone can sign up' kinda thing [15:30:49] RobH: sounds good [15:30:50] but, they are not happy, give messages like this with the current options [15:30:51] mount: 10.0.5.8:/home/wikipedia/wikistats: can't read superblock [15:30:57] but, if I add the nolock option [15:30:59] RobH: understand [15:30:59] it mounts [15:31:00] so would be like the board, the wikimania organizers who wanna see it, etc. [15:31:14] does it really need NFS? [15:31:16] and why? [15:31:21] mutante: so I wanted to test the gallium mysql templates but eventually gave up. I can't really have them deployed on labs. We will see that later next week when I am back from my looong weekend [15:31:25] we really hate NFS and are getting rid of it as much as possible [15:31:28] RobH: i think that works [15:31:29] good question, I am just trying to complete the ticket :) [15:32:08] I don't know what those remote NFS machines are [15:32:11] one is dataset2 [15:32:27] Erik Z uses this to generate some stats for the report card [15:32:32] we are going to replace this system eventually [15:32:35] but for now we are stuck with it [15:32:59] so it might be easiest just to get this running for now, until we get a new way of generating analytics data up ( > 6mo for sure) [15:33:25] nolock would be dangerous if anyone else is using the mounts, right? [15:33:35] either on the hosts or other remote nfs mounts [15:33:50] i doubt they are mounted anywhere other than bayes (deprecated) and stat1 (not yet) [15:34:00] so we could umount on bayes and use nolock… [15:34:08] or maybe if I umount on bayes I can mount on stat1 without nolock. [15:34:09] hm [15:34:17] will try that [15:34:34] but. in the meantime, do you what version of NFS those remote hosts are running? [15:34:48] if only /home/wikipedia/htdocs/wikipedia.org/wikistats is mounted [15:34:53] why don't we just move that directory onto stat1 then? [15:35:11] if nothing else mounts that dir, then there's no point in having NFS is there [15:35:19] and if something else does, locking is indeed a problem [15:35:21] there are 3 directories mounted, 2 from 10.0.5.8 and 1 from 208.80.152.185 [15:35:24] i think 208.80.152.185 is dataset1 [15:35:26] where xml dumps are stored [15:35:50] yeah [15:35:55] so the first one doesn't seem necessary [15:35:59] the second one is mediawiki [15:36:03] that might be necessary for something [15:36:15] and should probably be read-only? [15:36:28] hmm, maybe [15:36:35] the xmldump one probably for sure [15:36:39] yes [15:36:52] what machine is 10.0.5.8? what's it for? [15:36:54] mark, ottomate: i don't think we need NFS [15:37:00] 10.0.5.8 is /home [15:37:02] i think Erik just wants his data [15:37:15] can he just rsync it over? [15:37:16] sure, we should copy the existing wikistats directory off /home [15:37:19] I don't see why not [15:37:22] if nothing else uses that [15:37:28] so easiest is to see what data he uses / needs and copy it [15:37:30] and drop the mounts [15:37:32] indeed [15:37:42] yes, erik can rsync [15:37:44] if we copy /home/wikipedia over to stat1 and then have him work there from then on? [15:37:49] and just drop it elsewhere? [15:37:50] nono [15:37:52] not /home/wikipedia [15:37:52] no. [15:38:08] /home/wikipedia/htdocs/wikipedia.org/wikistats we can copy [15:38:11] ok [15:38:14] /home/wikipedia in its entirety we cannot [15:38:16] and /home/wikipedia/wikistats? [15:38:17] ok [15:38:20] so what else is needed there needs to be investigated [15:38:23] ok [15:38:29] yeah, i'm not sure exactly what he needs, it is a little confusing [15:38:35] we'll talk to him abou thte wikistats things then [15:38:38] i'm sure it is :) [15:38:38] as for dataset1 [15:38:48] i am waiting for erik to come online [15:38:50] can/should we still NFS that? [15:38:54] and i''ll ask him all that stuff [15:39:01] I can imagine that the dataset mount is needed [15:39:05] but I don't know what it's being used for [15:39:09] possibly just to READ data dumps [15:39:19] in that case it can become a read-only NFS mount [15:39:20] that's sounds likely [15:39:50] yeah probably [15:40:17] * apergos grits their teeth a bit [15:40:24] there's some pagecount stuff he munges [15:40:33] yes he does that as well [15:40:36] it may be that he'll want write for that [15:40:44] I don't remember how it's set up, even though I set it up :-/ [15:40:49] don't think so [15:40:52] but i'll ask him [15:40:57] so I shoud stab you for introducing another NFS mount then eh apergos [15:41:00] no [15:41:03] yes [15:41:09] hah [15:41:10] I didn't introduce this [15:41:17] you just said you set it up [15:41:19] ther eis a gluster copy of the most recent 5 dumps [15:41:21] maybe erik rsync thoses files [15:41:37] I would love it if he could use those ventually [15:41:51] we can setup now if that makes sense [15:42:00] the pageview stuff should go via rsync or whatever [15:42:08] indeed [15:42:24] I think we sohuld make that mount available to him (the gluster volume) and we'll find out what works and doesn't work [15:42:31] ok, still having trouble mounting dataset1 though [15:42:31] access denied by server while mounting 208.80.152.185:/data [15:42:34] even in ro [15:42:37] he's the first user so I expct we'll find various problems [15:42:43] sudo mount -t nfs -v -o ro 208.80.152.185:/data /mnt/data [15:42:52] apergos: can you work with them on that? [15:42:56] did you add yourself into the stanza in pppet? [15:43:06] the host, I mean [15:43:28] hmm, for export stuff? [15:43:29] so there's three steps. 1) puppet change for exports of dataset2 (not dataset1, it doesn't exist) [15:43:32] mark: can you approve https://gerrit.wikimedia.org/r/#change,5795 as well? [15:43:37] dataset2 aye [15:43:38] ok cool [15:43:40] will check [15:43:56] 2) puppet run on ds2 [15:44:12] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/5795 [15:44:15] 3) re-export ds2 (puppet can't do that right) [15:44:15] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/5795 [15:44:31] it can't? [15:44:33] thanks mark! [15:44:35] k, i don't have access to ds2 [15:44:43] re: NFS on stat1: https://gerrit.wikimedia.org/r/#patch,sidebyside,5709,1,manifests/misc/statistics.pp <-- there are the NFS mounts, added FIXMEs, the IPs and pathes are per ezachte from an RT [15:44:57] I'll do the puppet run and the re-export [15:45:05] thanks [15:45:16] add a big FIXME to get stat1 away from NFS there [15:45:19] <^demon> Do we have a generic wikimedia logo in files/ somewhere for placing on hosts? [15:46:42] apergos, does this look right? [15:46:43] http://pastebin.com/QcYj7h0m [15:47:01] adding stat1.wikimedia.org to that list? [15:48:03] uhh, hang on, that pastebin is not right [15:49:02] there [15:49:02] http://pastebin.com/P8GyaG8c [15:49:05] that's better [15:49:07]