[01:02:55] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=568571 edit summary: all are done [04:10:45] 08/03/2012 - 04:10:45 - Creating a home directory for burthsceh at /export/keys/burthsceh [04:15:47] 08/03/2012 - 04:15:47 - Updating keys for burthsceh at /export/keys/burthsceh [07:54:06] hello [07:55:24] yeahhh gluster 3!! [07:55:31] hashar@deployment-dbdump:~$ gluster --version [07:55:32] glusterfs 3.3.0 built on May 31 2012 10:40:41 [08:04:41] hashar is it better heh? [08:04:43] :P [08:04:53] petan: I need to test out git pull [08:05:01] andrewbogott_afk I suppose you didn't migrate bots yet [08:05:02] will update beta to latest master as well as all extensions [08:05:06] writing doc meanwhile [08:05:07] ok [08:05:18] hashar why do we even use gluster store for these files? [08:05:23] you moved the nfs to /data [08:05:24] ? [08:05:30] I thought we are going to use scap [08:05:39] also, I am not sure I told you, but I am on vacations for 3 weeks starting this evening [08:05:47] Reedy is taking backup [08:05:51] hashar I am on vacations 2 week starting sunday [08:05:57] cooool :-) [08:05:58] :D [08:06:21] so the reason for moving to /data was because nfs-memc instance added an another layer of abstration [08:06:34] and it started to be filling up heavily with all the files / thumbs [08:06:48] + we needed mooooar space for hosting transcoded video [08:07:07] that makes things a bit nicer / easier to understand too [08:10:39] http://en.wikipedia.beta.wmflabs.org/ : Forbidden! [08:10:40] yeahhh [08:11:28] !log deployment-prep Dist upgrading apache32 and 33 and rebooting them [08:11:30] Logged the message, Master. [08:12:18] I need to upgrade all instances [08:12:22] boo [08:12:42] !log deployment-prep dist-upgrading all instances to get the latest GlusterFS version (3.3.0) [08:12:44] Logged the message, Master. [08:13:47] petan: is deployment-feed still of any use ? I can't connect to it [08:13:59] hashar: yes it's for recovery [08:14:06] it contain some data that someone with shell need to recover [08:14:11] okkk [08:14:13] so that we can recreate it [08:15:07] oh that is https://bugzilla.wikimedia.org/show_bug.cgi?id=38611 [08:15:24] petan: can't we just redo the irc config? [08:20:27] petan: I am deleting deployment-nfs-memc , no more use for it, everything got copied / moved to /data/project [08:21:01] It has some weird irc config IIRC [08:26:03] we can't redo it because it's secret and on production [08:26:09] Ryan copied it to instance to /root [08:26:12] now it's corrupted [08:26:29] so I can't redo something I don't have access to [08:26:30] We should just puppetize it then push it up so prod can use it and labs can [08:26:34] [08:26:46] Damianz yes we wanted to do this for squid like 6 months ago, still waiting [08:27:00] Aren't we waiting for the squid configs from prod though? [08:27:04] yes [08:27:05] We actually had the irc config [08:27:05] still [08:27:10] we did [08:27:15] then it corrupted [08:27:18] now we don't [08:27:23] Not sure why it's THAT hard to get the squid configs [08:27:29] Smells like someone doesn't buy into progression [08:27:29] neither I do [08:27:49] I think there are some weird secret data, like Jimbo's secret bank account numbers in squid [08:27:50] Needs some ass kicking to the point that people stop pushing broken changes to prod [08:28:03] they just needed a place no one would look for them [08:28:10] squid was ideal [08:28:39] Anything secret can go in private anyway then just get included, like any block rules etc - don't need to know the ips blocked on prod, just the config option and add a list to secret [08:28:47] It's better for prod long term anyway [08:34:40] stupid Apache does not start :( [08:35:14] petan: squid is going to be phased out I think [08:35:20] I have done some work on using varnish [08:35:28] bits.beta is served by varnish iirc [08:35:37] hm [08:35:38] ok [08:35:40] good news [08:35:55] !log deployment-prep running puppet on Apache32 / 33 [08:35:57] Logged the message, Master. [08:36:28] ohhh [08:36:35] apache is disabled by default great [08:45:19] !log deployment-prep On -dbdump /etc/init.d/udp2log stop && /etc/init.d/udp2log-mw start {{bug|38995}} [08:45:21] Logged the message, Master. [08:48:48] !log deployment-prep making sure perms are correct in /data/project/apache/common-local/php-master : chmod -R g+w . ; chown mwdeploy:svn -R * .* [08:48:50] Logged the message, Master. [09:12:13] !log deployment-dbdump running git gc --aggressive in /home/wikipedia/common/php-master [09:12:13] deployment-dbdump is not a valid project. [09:12:22] !log deployment-prep running git gc --aggressive in /home/wikipedia/common/php-master [09:12:23] Logged the message, Master. [09:12:25] !beta [09:12:50] petan: would it be possible to create a !beta bot alias for !log deployment-prep ? ;-D [09:13:16] !log deployment-prep deleting deployment-bastion, it lacks a DNS entry {{bug|38846}} [09:13:18] Logged the message, Master. [09:29:56] Compressing objects: 32% (122457/373618) [09:29:57] sooo slow [09:49:30] !log deployment-prep On extensions: git submodule foreach git checkout master [09:49:31] Logged the message, Master. [09:53:43] !log deployment-prep Updating mediawiki core to latest master: Updating 0c1471c..d47c1e9 (fast forwarded) [09:53:45] Logged the message, Master. [09:54:01] hashar yes [09:54:15] !beta is !log deployment-prep $* [09:54:16] Key was added [09:54:20] !beta hashar :P [09:54:20] !log deployment-prep hashar :P [09:54:22] Logged the message, Master. [09:54:23] danke [09:54:37] !beta Updating extensions to latest master: git submodule foreach git pull [09:54:37] !log deployment-prep Updating extensions to latest master: git submodule foreach git pull [09:54:39] Logged the message, Master. [09:56:08] !* is $* [09:56:08] Key was added [09:56:19] !* a b c d e f g df g h er hj rtj t jr hs rg s gsd hse gdf dsf e e w tger g [09:56:19] a b c d e f g df g h er hj rtj t jr hs rg s gsd hse gdf dsf e e w tger g [09:56:21] ok [09:56:23] good [09:56:27] !! [09:56:27] an awesome bash trick [09:56:33] :O [09:56:34] !del !! [09:56:40] I think it's some alias [09:56:40] oh no [09:56:44] !! del [09:56:44] Successfully removed ! [09:56:49] !! [09:56:52] :P [09:56:58] I like doing double exclamations marks [09:57:04] !! [09:57:21] I am wondering what bash trick is that [09:57:31] ! is used to redo last command [09:57:32] Key was added [09:57:37] so I guess !! produce a loop [09:57:40] ! [09:57:40] used to redo last command [09:58:05] ! del [09:58:05] Successfully removed [09:58:12] ! [09:58:12] There are multiple keys, refine your input: $realm, $site, *, :), access, account, account-questions, accountreq, addresses, afk, alert, amend, ask, b, bang, bastion, beta, blehlogging, blueprint-dns, bot, bots, broken, bug, bz, console, cookies, credentials, cs, damianz, damianz's-reset, db, demon, deployment-beta-docs-1, deployment-prep, docs, documentation, domain, epad, etherpad, extension, forwarding, gerrit, gerritsearch, gerrit-wm, ghsh, git, git-puppet, gitweb, group, hashar, help, hexmode, hyperon, info, initial-login, instance, instance-json, instancelist, instanceproject, keys, labs, labsconf, labsconsole, labsconsole.wiki, labs-home-wm, labs-morebots, labs-nagios-wm, labs-project, leslie's-reset, link, linux, load, load-all, logbot, mac, magic, manage-projects, meh, monitor, morebots, nagios, nagios.wmflabs.org, nagios-fix, newgrp, new-labsuser, new-ldapuser, nova-resource, openstack-manager, origin/test, os-change, osm-bug, pageant, password, pastebin, pathconflict, petan, ping, pl, pong, port-forwarding, project-access, project-discuss, projects, puppet, puppetmaster::self, puppet-variables, putty, pxe, queue, quilt, report, requests, resource, revision, rights, rt, Ryan, ryanland, sal, SAL, security, security-groups, sexytime, socks-proxy, ssh, start, stucked, sudo, sudo-policies, svn, terminology, test, Thehelpfulone, unicorn, whatIwant, whitespace, wiki, wikitech, wikiversity-sandbox, windows, wl, wm-bot, [09:58:15] :P [09:58:19] !! [09:58:33] !say is $* [09:58:34] Key was added [09:58:37] !say Damianz hi [09:58:38] Damianz hi [09:58:45] !say hi | Damianz [09:58:45] Damianz: hi [09:58:47] :D [09:58:53] !say IMMA EAT YOUR BRAINS | petan [09:58:54] petan: IMMA EAT YOUR BRAINS [09:58:56] !say how are you, dude | Damianz [09:58:56] Damianz: how are you, dude [09:59:19] On another note, going back to writing python :P [09:59:21] !say !say hello | wm-bot | wm-bot [09:59:21] wm-bot | wm-bot: !say hello [09:59:43] !sayloop is !say hi [09:59:43] Key was added [09:59:45] can't look it [09:59:46] !sayloop [09:59:46] !say hi [09:59:54] doesn't consider itself as a user [09:59:58] !sayloop del [09:59:58] Successfully removed sayloop [10:00:06] !sayloop2 is !sayloop2 [10:00:06] Key was added [10:00:09] !sayloop2 [10:00:09] !sayloop2 [10:00:16] it's idiot proof [10:00:18] :D [10:00:23] ahh it is smart enough to not react to itself [10:00:37] !sayloop2 del [10:00:37] Successfully removed sayloop2 [10:01:37] petan: since you like bots, would you be interested in setting up a bugzilla bot to replace wikibugs ? [10:01:49] Mozilla / Gnome have a nice one that comes as abugzilla plugin [10:01:54] ok [10:02:02] http://code.google.com/p/supybot-bugzilla/ [10:02:05] too many bots [10:02:09] that would be an interesting project for the bots labs [10:02:11] but I need to know how it's gonna work [10:02:25] well one would need to create a new instance, install bugzilla [10:02:29] Danny_B|backup I can make it a plugin for wm-bot [10:02:37] + install supybot and write doc about how to do so [10:02:41] then play with it here :-) [10:02:56] that would prepare us to install supybot in production and phase out wikibugs [10:03:10] hashar: once I get some information what that bot should do I can just make a plugin [10:03:25] or is it needed for it to be supybot? [10:03:42] because I really don't like python [10:03:57] I hate languagues which parse code by indentation [10:04:22] which is there on purpose to improve the code legibility [10:07:23] ok I am just used to read code by brackets and when I see no brackets it's like if it was whole one function to me [10:07:38] !beta running massive git gc --aggressive on all extensions [10:07:38] !log deployment-prep running massive git gc --aggressive on all extensions [10:07:40] Logged the message, Master. [10:07:50] git garbage collector? [10:08:06] :P [10:08:06] petan: the proposal is to test out supybot :-) [10:08:13] that is actively maintained by an upstream [10:08:23] ok, as long as I don't have to modify the source code, I am fine with that [10:08:26] so guarantee to be better out of the box :D [10:08:29] * Danny_B|backup thinks bots should be written in non compiled langs [10:08:42] * petan disagrees [10:08:50] * Danny_B|backup knows ;-) [10:08:54] petan: well it is more about trying to install it and write some short report about what it is [10:08:56] the good points [10:08:59] the ease of installation [10:09:00] aha [10:09:04] well I can test it [10:09:08] well overall, just trying out supybost :) [10:09:11] c# is sort of far away from open source stuff though [10:09:12] ok [10:09:18] see if it could be a nice candidate for wikibugs replacement [10:09:21] Danny_B|backup it's a new language [10:09:26] hopefully it will change in future [10:09:31] mono is open source [10:09:32] fuck [10:09:35] I killed -dbdump [10:09:37] damn [10:09:38] o.o [10:09:43] poor box [10:09:47] git gc --aggressive with 4 threads sent the box to swap I guess [10:09:48] haha [10:09:50] :-( [10:09:55] !beta hashar is going to kill us all [10:09:55] !log deployment-prep hashar is going to kill us all [10:09:57] Logged the message, Master. [10:10:14] !beta Probably killed -dbdump by launching 4 instances of git gc --aggressive :-((((((((((( [10:10:14] hashar we should disable swap [10:10:14] !log deployment-prep Probably killed -dbdump by launching 4 instances of git gc --aggressive :-((((((((((( [10:10:16] Logged the message, Master. [10:10:21] it suck [10:10:28] on system with poort IO like labs [10:10:30] poor [10:10:50] swap == terminated [10:10:57] hehe [10:11:03] Sarah Connors ? [10:11:03] if we disable swap it at least die on OOM [10:11:53] !beta rebooting swapped -dbdump [10:11:54] !log deployment-prep rebooting swapped -dbdump [10:11:55] Logged the message, Master. [10:11:55] btw why linux die on OOM so ungracefully compared to windows [10:11:58] swapdeath ftl [10:12:04] windows never die on OOM [10:12:11] it just kill the low priority processes [10:12:22] yeah stuff like mysql [10:12:23] ;) [10:12:26] heh [10:12:27] it happened to us in production [10:12:37] I think one time it killed the master DB :-] [10:12:45] with a -9 signal!!!!!!!!!!!!!!!!!!!!! [10:13:03] caused several hours of downtime while the db was being reconstructed out of log files [10:13:08] and we did lost some transactions :-( [10:14:27] once it happened that typo (non-paired parenthesis) in template killed whole server farm so there was wikimedia blackout for about three hours [10:14:44] haha [10:15:21] I am back, danny crashed my client :D [10:15:32] because it's written by me :P [10:15:39] what? [10:15:55] oh if you guys are on linkedIn, I have created a "We are Anonymous" group to support the cause ;) http://www.linkedin.com/groups/We-are-Anonymous-4557326 [10:16:28] being a member of any group means i am no longer anonymous ;-) [10:17:51] ok I am back again [10:17:57] my client didn't like the | symbol [10:18:02] hopefully it works now [10:18:06] I lost context a bit [10:18:51] Danny_B|backup do you see my responses or private messages are broken as well [10:18:55] Oops | [10:19:04] Damianz in nickname [10:19:10] but it's fixed now [10:20:59] !beta running 'git submodule foreach git gc --aggressive' in screen 2042 [10:20:59] !log deployment-prep running 'git submodule foreach git gc --aggressive' in screen 2042 [10:21:01] Logged the message, Master. [10:21:09] hashar, should I create a project for that [10:21:10] bz [10:21:18] mutante hi [10:21:26] petan: well just create an instance in the bots project ? [10:21:30] I guess that will be enough [10:21:36] hashar we should install a bugzilla to test it? [10:21:42] and this way other members of the bots project can play with it too :) [10:21:46] I don't like idea of installing bugzilla to bots project [10:21:51] ohh [10:21:58] then create a new project [10:22:02] maybe we already have some in labs [10:22:03] maybe there is a bugzilla one [10:22:06] there is project bugzilla [10:22:10] dunno what is there [10:22:15] no idea either [10:22:15] neither who maintains it [10:22:19] mutante are you here :D [10:22:22] I guess it was hexmode [10:22:32] !say I need you to insert me to bugzilla project | mutante [10:22:32] mutante: I need you to insert me to bugzilla project [10:22:35] mutante is in SF I think [10:22:40] aha [10:22:58] no hexmode here [10:23:00] :o [10:23:03] a dedicated instance in bots project will save you the trouble of adding new users I guess [10:23:04] he left irc... [10:23:09] well it is up to you :-] [10:23:20] I don't want to mix bots with other project [10:23:39] I will run the bot on bots, but I don't want to ahve bugzilla there [10:23:50] some generic testing bugzilla would be nice [10:24:26] Hey dawg, I heard you like bugzilla so I put bugzilla in your bugzilla so you can report while you report. [10:35:42] !log incubator Switching both enwiki and devwiki to the master branch in the name of logic (aka to really do testing and not be stuck with 1.20wmf* branches). [10:35:43] Logged the message, Master. [11:11:15] !log dumps Having persistent problems with accessing the gluster storage, grrr... [11:11:16] Logged the message, Master. [11:25:47] 08/03/2012 - 11:25:46 - Creating a home directory for matmarex at /export/keys/matmarex [11:30:47] 08/03/2012 - 11:30:47 - Updating keys for matmarex at /export/keys/matmarex [12:39:31] Change on 12mediawiki a page Developer access was modified, changed by Planetenxin link https://www.mediawiki.org/w/index.php?diff=568701 edit summary: planetenxin [13:36:08] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=568710 edit summary: archived some reqs [13:37:42] !beta cache-bits02 : cd /var/lib/git/operations/puppet sudo GIT_SSH=/var/lib/git/ssh git fetch origin refs/changes/04/13304/14 && git checkout -b 13304/14 FETCH_HEAD (aka deploying {{gerrit|13304}} patchset 14 [13:37:43] 13304}} patchset 14: !log deployment-prep cache-bits02 : cd /var/lib/git/operations/puppet sudo GIT_SSH=/var/lib/git/ssh git fetch origin refs/changes/04/13304/14 && git checkout -b 13304/14 FETCH_HEAD (aka deploying {{gerrit [13:38:00] !log deployment-prep cache-bits02 : cd /var/lib/git/operations/puppet sudo GIT_SSH=/var/lib/git/ssh git fetch origin refs/changes/04/13304/14 && git checkout -b 13304/14 FETCH_HEAD (aka deploying {{gerrit|13304}} patchset 14 [13:38:02] Logged the message, Master. [13:38:04] ;) [13:38:12] petan: the !beta alias does not work all the time hehe [13:38:21] yes, pipes do not [13:38:28] it's hard to make it do both things [13:38:43] people want to use pipes [13:38:44] !beta switching cache-bits02 from role::cache::bits::labs to role::cache::bits ( see {{gerrit|13304}} patchset 14 [13:38:44] 13304}} patchset 14: !log deployment-prep switching cache-bits02 from role::cache::bits::labs to role::cache::bits ( see {{gerrit [13:38:56] {{gerrit|blah [13:39:00] is problem [13:39:17] when you use a pipe bot use the string after is as a prefix for output [13:39:21] !say hi | hashar [13:39:21] hashar: hi [13:39:25] you see [13:39:36] !say {{gerrit|13 [13:39:36] 13: {{gerrit [13:40:29] hashar you either need to avoid pipes or I would need to put the bots to another channel, because here we need to use that pipe thingy [13:42:25] petan: na that is fine :-° [13:42:41] I will just fall back to the good old """ !log deployment-prep """ :-) [13:42:46] the beta alias is nice enough [13:47:01] !log deployment-prep deployed {{gerrit|13304}} PS 14 on cache-bits02 [13:47:03] Logged the message, Master. [14:13:16] paravoid: have you found any inspiration to get syslog-ng on deployment-dbdump ? :-D [14:15:46] 08/03/2012 - 14:15:46 - Creating a home directory for planetenxin at /export/keys/planetenxin [14:20:48] 08/03/2012 - 14:20:47 - Updating keys for planetenxin at /export/keys/planetenxin [14:25:47] 08/03/2012 - 14:25:47 - Updating keys for planetenxin at /export/keys/planetenxin [15:20:50] andrewbogott: I'm very impressed [15:20:57] how much time did it take you to do all these migrations? [15:21:21] (good morning) [15:21:27] Yesterday was lots of meetings so it was easy to keep half an eye on the migration terminal. [15:21:40] And there weren't all that many left anyway [15:21:44] good morning! [15:22:49] I've been expecting a dozen people to appear in this channel and ask "why doesn't my instance work anymore?" But so far things seem peaceful. [15:23:06] yeah :) [15:23:19] petan has sent me an email on how to restart some of the bots [15:23:39] Is he not going to be around to do it himself? [15:23:48] apparently no [15:23:56] I just fwd you the mail, I can help with that too [15:24:13] ok, I see it. [15:31:33] !log deployment-prep updating cache-bits02 puppet to gerrit 13304/21 [15:31:35] Logged the message, Master. [15:53:07] paravoid: andrewbogott: i could maybe attempt to help restarting too [15:53:14] is it scheduled for a specific time? [15:53:27] a little after 18 UTC? [15:53:30] I'm going to start breaking things at 11pdt. [15:53:45] But it'll take ~30 minutes to move all the bots instances. [15:53:55] 11pdt = 18utc, I hope? [15:53:59] yes [15:55:34] jeremyb: I think the known set of things to restart is small; I'm not sure what to do about the unknown set except to let the bot-herders take care of their own. [15:55:49] * andrewbogott wonders if 'bot herder' is an epithet [15:56:48] andrewbogott: we could maybe make a process listing for each box before the boot to have a hit list of things to make sure are up after? [15:57:07] !log deployment-prep cache-bits02 would need 13304 to be merged in manually whenever the change is completed. Make sure to report any issue to mark :-] [15:57:09] Logged the message, Master. [15:57:11] I am off for vacations [15:57:16] ii [15:58:29] jeremyb: That's a good idea, although it sounds like a lot of work [15:58:45] andrewbogott: idk... [16:09:15] jeremyb: I think a translation of 'sounds like a lot of work' is: I would be happy for you to do this but would not be happy to do it myself :) [16:09:34] ldapsearch -v -x -y <(< /etc/ldap.conf awk '$1 == "bindpw" { print $2}' | perl -pe 's/\s//g;') -D 'cn=proxyagent,ou=profile,dc=wikimedia,dc=org' -b 'ou=hosts,dc=wikimedia,dc=org' 'puppetVar=instanceproject=bots' puppetVar 2>/dev/null | fgrep 'instancename=' | awk -F= '{print $2".pmtpa.wmflabs"}' [16:09:57] that's a list of hostnames that i should get process lists for [16:10:24] I would've gotten the list from nova, but it's probably the same list. [16:11:14] * aude reads that as wikimediadc.org ;) [16:11:56] aude: ;) [17:00:45] 08/03/2012 - 17:00:44 - Updating keys for burthsceh at /export/keys/burthsceh [17:01:45] REMINDER: There will be a brief bastion outage in one hour. Anyone who is logged into a labs instance at that time will be unceremoniously kicked off. [17:13:59] j^: what's wrong with it? how can I help? [17:16:18] yeah, looking [17:16:43] I was looking at your jobrunner stuff just now [17:16:57] I feel terrible for taking so long, lots of work this week [17:17:37] why did I remember that mark reviewed that at some point? [17:22:35] j^: is upload better now? [17:22:37] should be [17:24:42] grat [17:24:44] great* [17:48:54] REMINDER: There will be a brief bastion outage in ten minutes. Anyone who is logged into a labs instance at that time will be unceremoniously kicked off. [17:56:22] paravoid: I created a new instance this morning and it landed on virt1. Is that expected? [17:56:35] unfortunately yes [17:56:52] according to Ryan, the scheduler picks the emptiest node [17:57:01] which means that during the transition there's no way to fix that [17:57:07] but after a node is empty, we can stop nova-compute on it [17:57:27] ok [18:00:16] andrewbogott: warn me when you're starting [18:00:31] jeremyb: I'm starting! [18:00:56] That goes for everyone: bastion instances are about to reboot, so hit save now! [18:02:09] well i got the ps just in time ;) [18:04:11] Heh, I was tunneled into IRC via bastion [18:04:19] hahaha [18:05:28] * Damianz paws bogott [18:05:47] jeremyb: Can you reach bastion1 now? [18:05:52] i'm on it [18:06:06] jeremyb@bastion1:~$ uptime 18:06:01 up 0 min, 1 user, load average: 0.23, 0.06, 0.02 [18:06:11] OK, great, bastion hosts are moved. [18:06:17] So, now, time to break bots. [18:06:43] andrewbogott: getting some food, back in ~20 mins [18:07:43] On the bright side my supervisord setup works from last time Ryan broke bots :D [19:04:07] when the bot dies on a !log command it was usually the missing cache directory [19:04:09] (or write perms to it) [19:05:25] mutante: I fixed that in the new version [19:05:41] we need to delete the bot off of the old box [19:07:00] cool [19:07:43]