[01:42:09] Thehelpfulone: hey. let me take a look [01:42:24] although -> #-labs [01:43:22] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 388 seconds [01:45:46] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [03:51:10] RECOVERY - Puppet freshness on spence is OK: puppet ran at Mon May 21 03:50:41 UTC 2012 [04:15:53] PROBLEM - udp2log log age for emery on emery is CRITICAL: CRITICAL: log files /var/log/squid/orange-ivory-coast.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [04:44:30] Reedy: still around? [04:46:06] i wonder if stat1 ever got the reinstall? there should be an RT on it [04:46:12] May 11: 11:35 mutante: stat1 - installed new kernel, but waiting to reboot. schedule with aotto [04:46:19] but nothing more recent [04:48:34] * jeremyb tries to decide if he's reading this right [04:48:43] stat1 has nearly 10TB free? [04:49:00] and not much more than that total? [04:49:05] (ganglia) [04:50:43] also seems to have a lot of ram (30ish GB?) and CPU (16 cores?) [04:51:27] anyway, i see no obvious issues in ganglia but i don't really know what to look for [04:51:48] http://stats.wikimedia.org/EN/TablesWikipediaEN.htm is up to date through april (the top row of the table [04:51:57] http://stats.wikimedia.org/EN/TablesWikipediaID.htm is only through march [04:52:16] same with Summary${lang}.htm [04:56:00] hrmm... so those files are derived from apergos's DB dumps apparently [04:56:30] anyone have a TZ on ezachte? [05:01:20] hrmm, i wonder if it's just crond not running? [05:02:03] (how often do these pages get updates? they all seem to say they were last updated on the 16th. maybe that's normal or maybe it's frozen in time) [05:02:40] but puppet ran recently... [05:04:35] i don't see any sign that puppet ensures crond is running [05:04:41] (in the repo) [05:04:54] * jeremyb has quite a monologue going! ;P [05:17:15] RECOVERY - udp2log log age for emery on emery is OK: OK: all log files active [06:03:58] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [06:34:16] ok, see wikitech-l for same issue as above [06:44:40] also, gerrit-wm still needs a boot. (post #mediawiki banning) [06:47:05] bye [06:47:18] bye [06:47:26] sleep!? [06:47:34] maybe! [06:48:16] * aude eating breakfast [06:48:25] {{fact}} [06:48:33] :) [07:43:28] helllo [08:12:38] New patchset: Dzahn; "add detail page for a single wiki and link in id column globally" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8299 [08:12:39] New patchset: Dzahn; "add pages / offset to display.php, get rid of old lines / threshold, shorten sort= to s=,..." [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8300 [08:12:40] New patchset: Dzahn; "remove timestamp colors from code and into css classes, add the css file, add "..hours ago" to ts column," [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8301 [08:12:41] New patchset: Dzahn; "retab css file, add classes for version colors, make timestamp oldness configurable,..." [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8302 [08:12:41] New patchset: Dzahn; "adjust require pathes, use $IP, add required coalesced_query" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8303 [08:12:42] New patchset: Dzahn; "enhance ugly sortswitch, get rid of undefined constant / index warnings" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8304 [08:12:43] New patchset: Dzahn; "remove sorting stuff from detail page, move table creation and more into functions, add CSS, fix invalid HTML,..." [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8305 [08:14:00] New patchset: Dzahn; "adjust device names / partition sizes in partman for analytics" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8306 [08:14:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8306 [08:21:03] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8299 [08:21:05] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8299 [08:21:53] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8300 [08:21:55] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8300 [08:23:02] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8301 [08:23:04] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8301 [08:23:54] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8302 [08:23:55] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8302 [08:24:49] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8303 [08:24:50] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8303 [08:25:44] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8304 [08:25:45] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8304 [08:26:58] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8305 [08:27:00] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/8305 [08:27:51] New review: Dzahn; "Ciscos use sdc/sdd" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8306 [08:27:53] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8306 [08:29:36] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [08:29:55] New patchset: ArielGlenn; "specific basedir as option to make deployment easier" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/8309 [08:30:32] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8309 [08:30:34] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/8309 [08:31:36] Change abandoned: Dzahn; "there should be better ways and meanwhile it is running on all anyways" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3510 [08:33:24] New review: Dzahn; "puppetize logrotate config from brewster and change to size based rotation" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7167 [08:33:27] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7167 [08:56:36] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [09:09:37] New review: QChris; "Is the documentation of --basedir consistent with the way CONFIGFILE is used?" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/8309 [10:05:17] New patchset: Jeroen De Dauw; "add irc notifications for the #wikimedia-wikidata and #semantic-mediawiki channels" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8043 [10:05:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8043 [11:36:37] PROBLEM - Host analytics1002 is DOWN: PING CRITICAL - Packet loss = 100% [11:38:31] ACKNOWLEDGEMENT - Host analytics1002 is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn reinstall [11:40:40] RECOVERY - Host analytics1002 is UP: PING OK - Packet loss = 0%, RTA = 26.43 ms [12:26:31] New review: Nikerabbit; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/8043 [12:29:14] PROBLEM - SSH on virt2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:35] RECOVERY - SSH on virt2 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:39:23] * jeremyb scrolls up [12:39:40] no labs? [12:39:56] hashar: could you boot gerrit-wm ? (post #mediawiki banning) [12:40:07] ? [12:40:15] wtf are people banning gerrit-wm ? [12:40:30] talk to Joan or see logs [12:40:34] hashar: yes [12:40:57] who ever banned it deserve to get ops right removed [12:41:02] unless it suddenly became wild [12:41:02] 21 02:51:43 < jeremyb> damnit joan [12:41:03] 21 02:52:02 < jeremyb> 20 19:53:35 -!- gerrit-wm was kicked from #mediawiki by Joan [gerrit-wm] [12:41:07] ... [12:41:11] it became wild [12:41:15] k k [12:41:32] * aude hears [12:41:32] anyway [12:41:32] I am pretty sure I don't have ops access there [12:41:34] i think maybe it was someone using that script to do a mass approave or something [12:41:49] oh I do [12:41:49] hashar: i'm saying boot the process on manganese [12:43:45] hashar: it's currently a +q (p858 changed from Joan's +b) [12:43:59] I don't have access to manganese [12:44:02] oh [12:44:04] ;-( [12:45:51] !log Started ircecho on manganese [12:45:56] Logged the message, Master [12:46:05] our hero :-D thanks mark [12:47:33] New patchset: Thehelpfulone; "flood group to enwikisource, bug 36863" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/8326 [12:47:33] New patchset: Thehelpfulone; "flood group to enwikisource, bug 36863" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/8326 [12:47:42] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/8326 [12:47:42] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/8326 [12:47:43] hmm double bot? [12:48:28] mark: could we have just one copy? ;-) [12:48:50] the more the merrier? [12:48:51] ;) [12:48:57] :P [12:49:04] mark: ask Krinkle about that [12:49:19] * jeremyb incorporates dbbot-wm's history by reference [12:50:29] jeremyb: Ask me about what? Confused dbbot-wm with gerrit-wm ? [12:50:42] Krinkle: "the more the merrier" [12:51:10] heh :) [12:59:10] New review: Nikerabbit; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/8037 [13:06:28] Dunno if anyone is around that knows labs stuff, but labs seems to have deadful I/O issues today to the point of not being able to login to servers. If you can do anything about it, awesome, if not Ryan will get badgered later. [13:06:44] Damianz: i think people know [13:07:11] jeremyb: Never assume! Or I just like annoying people.... not sure which [13:07:22] and have done some diagnosis. haven't caught up yet on what exactly [13:07:34] Damianz: sure. i'm just advising you that you should assume in this case [13:08:23] Considering people have been moaning since this morning and there's numerous bz tickets open I'd hope so - looking at how long a review takes, who knows though :P [13:09:59] Damianz: I'm here and have access but I'm not really sure what to do :-) [13:10:05] trying to get the hang of it still [13:10:26] ahhhh [13:10:29] GIT [13:10:36] I'd sugest gluster is being a tool but apart from that who knows without poking, Ryan still seems to be the person who knows the most sadly [13:10:40] I am on a project still using svn [13:10:41] noo [13:11:15] * jeremyb waves a ^demon [13:11:20] just in time [13:11:26] <^demon> uh oh [13:11:31] * hashar opens a bug about migrating all /trunk/tools project from svn to git [13:11:34] \o/ [13:11:36] good morning Chad [13:11:52] * jeremyb annexes richmond to DC [13:12:11] * hashar uses `git svn clone` [13:12:11] <^demon> annexing doesn't make it closer. [13:12:25] <^demon> hashar: We can just write rules for svn2git and do them all at once. [13:13:32] don't you already have rules to do that ? [13:13:34] New review: Nikerabbit; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/5778 [13:13:38] oh forget me [13:13:46] <^demon> For projects we've already converted ;-) [13:13:56] yeah rules for svn2git will be great, isn't it all about adding a new path in some conf file ? [13:15:35] New review: Demon; "(no comment)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/8120 [13:17:23] ^demon: isinstance? what's the question? [13:17:51] <^demon> You typo'd it? [13:17:52] * jeremyb points to http://docs.python.org/library/functions.html#isinstance [13:17:59] nope [13:18:00] <^demon> if isinstnace [13:18:06] <^demon> instnace? [13:18:08] no [13:19:23] <^demon> On line 138 you used "isinstance". On line 134 its "isinstanace" [13:19:36] <^demon> *isistnace [13:19:44] ohhh [13:20:02] well i typed it right just now [13:20:04] ;P [13:20:06] <^demon> :) [13:20:07] * jeremyb rereads [13:20:24] but otherwise? [13:20:26] <^demon> hashar: Here's a start http://etherpad.wikimedia.org/Git-migration-tools [13:20:40] <^demon> jeremyb: *looks* ok, but I haven't cherry picked it and tested yet. [13:21:31] ^demon: that is all cryptic to me ;-D [13:21:39] ohh [13:21:50] <^demon> The only part you'd need to add is more create repository lines [13:21:58] <^demon> The match is a regex :) [13:22:03] well I am not sure we want to do a brute force migration of /trunk/tools/* to some tools/$1 git repository [13:22:15] some projects there are obsolete [13:22:35] <^demon> Well it's harmless to dump them all. If we don't want them in gerrit just don't make the projects there. [13:22:50] also I was wondering this morning if we really want a new namespace such as tools/* [13:23:00] which is not very meaningful [13:23:00] <^demon> I already added mediawiki/tools/* [13:23:04] kk [13:23:04] <^demon> which some of that will go into [13:23:49] that was my question on some bug requesting to migrate the multiversion script [13:24:15] my proposal is operations/hetdeploy or operations/mediawiki/multiversion [13:24:25] or could be some platform/mwmultiversion [13:24:37] I have really no idea and can't make a choice :-( [13:25:04] <^demon> operations/* makes sense for it. We can bikeshed over the actual repo name later. [13:25:44] sure [13:25:58] wanna review the various tools ? [13:26:22] <^demon> I can this afternoon. I've got class in an hour and need to leave in ~30m. [13:27:57] New patchset: Jeremyb; "cleanup/refactor gerrit logging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8120 [13:28:16] ok [13:28:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8120 [13:28:28] thanks gerrit2 ;) [13:32:16] huh. `git reset --hard origin/production` and then `git pull -a` fails with [13:32:19] Unable to find common commit with 1700114759097cb2237aa9e6e332093a90021926 [13:32:36] which is a commit from december [13:41:29] this is getting annoying [13:41:57] i deleted my local production and then did `git checkout -t origin/production` [13:42:08] then git pull -a gives same error again [13:43:57] PROBLEM - Host analytics1002 is DOWN: PING CRITICAL - Packet loss = 100% [13:43:59] huh, i don't have a clue how i fixed it. ;-( [13:44:01] $ git pull -a [13:44:01] Already up-to-date. Yeeah! [13:47:52] New review: Jeremyb; "(no comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8120 [13:49:54] New review: Demon; "Testing claimed commenting bug." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8120 [14:11:41] RECOVERY - Host analytics1002 is UP: PING OK - Packet loss = 0%, RTA = 26.38 ms [14:37:04] cmjohnson1: ping [14:37:10] would you have time to look at the netapp? [14:46:56] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: CRIT replication delay 184 seconds [14:47:32] PROBLEM - MySQL Replication Heartbeat on db1018 is CRITICAL: CRIT replication delay 192 seconds [14:51:44] RECOVERY - MySQL Replication Heartbeat on db1018 is OK: OK replication delay 16 seconds [14:52:38] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [14:53:19] hi mutante, did CT talk to you about getting us sudo on the new analytics box(es)? [14:55:52] aaannd mark, if you are around and have a sec sometime soon [14:55:54] https://gerrit.wikimedia.org/r/#/c/7988/ [14:56:12] would love to review that with you (or someone), so that if you have questions I can answer [14:56:32] you seem to be the most appropriate review, since you reviewed my git stuff too [15:04:06] mark: heya [15:04:10] hi [15:04:22] I can't get one of the two netapp controllers to respond [15:04:24] I am in eqiad w/ rob this week [15:04:27] oh [15:04:28] right [15:04:30] meh [15:04:57] how do you like eqiad? [15:05:03] much nicer than tampa! [15:05:10] no shit ;) [15:06:38] put a ticket in and I will get to it when I get back [15:06:46] yeah [15:11:23] ottomata: coming up within the next hour or so. 8/10 installed ..;) [15:12:24] oh amazing! [15:12:26] thanks! [15:12:49] ottomata: speak of the devil ;) [15:13:18] mutante is the devil!? [15:13:27] * mutante hides [15:13:32] dzahn == DEVIL ZAHN!? [15:13:40] I was going to say woosters :P [15:13:45] ohhhh [15:13:58] but sure, mutante sounds like a devilish nick too [15:14:25] it's just biology :p [15:55:52] New patchset: Dzahn; "full sudo privs for diederik,dsc and otto on analytics hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8336 [15:56:12] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8336 [15:57:17] Q about sudo privs [15:57:23] why do we do NO PASSWORD? [15:57:25] New review: Dzahn; "per CT" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8336 [15:57:27] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8336 [15:57:59] ottomata: because you seemed to want it on others and copying [15:58:37] ottomata: happy to change it to regular [15:59:40] you had it on all existing ones though [16:00:07] hmm, [16:00:11] * jeremyb personally prefers NOPASSWD [16:00:17] i don't mind, but I think it is safer to have to enter our sudo password when we sudo [16:00:49] we can do another commit to change all at once then if you want to change it, staying consistent for the moment [16:01:04] what do other ops people do? is everyone usually NOPASSWD? [16:01:35] well, they are root [16:01:57] ? [16:02:04] they sign in as root? [16:03:13] you have all 3 groups, people who can root, people who have sudo ALL, and people with selected sudo commands [16:03:28] where i see ALL i also see NOPASSWD combined with it though [16:03:34] hmm, ok [16:03:39] well i guess let's just be consistent, i was just wondering [16:04:41] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [16:07:07] ottomata: feel free to test on analytics1006 [16:07:43] it already created sudoers.d files for each user there [16:14:24] it works! [16:14:33] :) [16:15:25] ottomata: expect all to work except 1005 (1001 to 1010) [16:15:39] that's the last one? [16:15:48] yes, 1005 is rebooting [16:16:59] you got 2 software raid 1, one for / and one for swap, across first 2 disks (which are sdc and sdd) [16:17:32] RobH: do we have any spare misc servers in pmtpa? we may need one for labs [16:17:35] /dev/md0 being mounted as / [16:17:59] perfect, thanks so much! [16:19:11] ok, let me sign 1005 on puppetmaster and we should be fine for now [16:22:29] !log analytics1001 to 1010 installed and up in puppet [16:22:31] paravoid: labs cannot run on normal misc stuff [16:22:33] Logged the message, Master [16:22:42] paravoid: the ciscos will be shipping there this week for install next and the following [16:24:11] RobH: we were talking about using that as a temporary NFS server, so there's no need for a Cisco for that [16:24:28] RobH: a server with somewhat acceptable I/O would be fine [16:24:49] nfs server as opposed to the gluster servers dedicated to that kinda thing? [16:25:13] i may have a couple of spare misc servers, i would have to do a review to find out [16:25:17] but those servers dont have a lot of storage [16:25:25] misc servers tend to be dual 250gb [16:25:25] nfs server as opposed to an instance that runs an nfs server that in turn uses gluster [16:25:41] we have 20g right now iirc, so it should be fine [16:26:25] yep, 18g allocated, 12g used [16:27:11] ok, if ya need it go ahead and drop a ticket in procurement listing what you need and why you need it pls [16:27:17] that way i have a trail of whats goin on ;] [16:27:25] sure [16:27:45] should I file it in procurement even though we don't actually want to procure anything? [16:27:59] if we don't have any spares, we shouldn't buy one, since this should be a temporary measure [16:31:08] Ryan_Lane: are you okay with that plan? should I go ahead and file the ticket? [16:35:50] gimme a sec to read backscroll [16:36:03] I'm in the middle of an IRC onslaught [16:36:11] heh [16:37:11] buying one will take longer than using the ciscos [16:37:25] if we have an available misc, that'll work [16:37:34] otherwise let's plan on riding out the storm [16:37:48] agreed [16:47:35] New patchset: Dzahn; "add monitoring group analytics-eqiad and set in analytics role class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8338 [16:47:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8338 [16:50:35] New patchset: Hashar; "puppetize wikibugs (irc bot for bugzilla)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8339 [16:50:55] New review: Dzahn; "this time with a DC suffix, ack" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8338 [16:50:55] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8338 [16:50:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8339 [16:58:36] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7884 [16:58:39] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7884 [17:01:16] thanks ben! [17:16:17] ottomata: you're welcome! but it's not live yet, because puppet takes frikkin hours on spence. [17:16:26] ha, ok [17:17:11] maplebed: did you do anything special to the sqlite dbs in swift? [17:18:07] AaronSchulz: no. [17:18:15] why, are they special? [17:18:27] it easily takes 15 seconds to list some thumbnails [17:18:37] in fact that's where all the slowness is [17:18:49] the concurrent deletes are very fast, even the 99th percentile [17:18:57] fantastic! [17:19:12] * AaronSchulz looks at db.py [17:19:14] CREATE INDEX ix_object_deleted_name ON object (deleted, name); [17:19:37] maplebed: just for sanity can you check if that index is there? [17:19:42] I never expected listings to be super quick, but 15s is a little absurd. [17:20:07] I don't know how to use sqlite very well. I'll try and find out. [17:20:29] and no we can't switch the DBMS, so says launchpad ;) [17:20:43] * AaronSchulz wonders what version of sqlite is running [17:21:04] 3.6.22-1 [17:21:15] ^^^ the package version for libsqlite3-0. [17:21:36] New patchset: Ottomata; "Creating generic::mysql::server class that installs packages and sets up my.cnf and starts mysqld." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7988 [17:21:38] burrito time. [17:21:55] Change abandoned: Ottomata; "Including this change in this commit:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7887 [17:21:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/7988 [17:22:42] hiyaaaaaa, mark, are you still around? [17:23:00] or maybe someone else who's got puppet / mysql opinions? [17:48:11] !log ran authdns-update on dobson to pick up virt1002-1008 changes [17:48:16] Logged the message, Master [17:51:44] New patchset: Jeremyb; "make ircecho config sane (not just very long strings)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8344 [17:52:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8344 [17:52:04] New review: Jeremyb; "(no comment)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/8043 [18:04:45] !log dist-upgrade and reboot loudon [18:04:48] Logged the message, Master [18:23:16] ottomata: 3 diffs between your method and ^demon's. that i can think of off the top of my head. 1) name translation varies (no @users.mediawiki.org in yours) 2) svn's not made read only so people don't accidentally commit there and 3) ^demon maybe doesn't know it has happened [18:24:06] prolly other diffs with the commit msgs [18:27:17] uhh, wait, what are we talking about? [18:27:20] <^demon> You miss the refs/notes/commits that has svn commit data. [18:27:23] <^demon> jeremyb: ^ [18:27:40] New patchset: Reedy; "Added php-htmlpurifier to go with https://gerrit.wikimedia.org/r/#/c/7826/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8348 [18:28:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8348 [18:30:21] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [18:31:02] ottomata: your svn -> git mail [18:31:24] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [18:31:37] ottomata: want to review 8344? ;) [18:32:02] oh about the move from svn to udplog [18:32:02] ? [18:32:07] sorry [18:32:08] yeah [18:32:11] svn udplog to git udplog [18:32:20] still yearh [18:32:20] ^daemon == chad? [18:32:21] yeah* [18:32:25] yes! [18:32:37] how did you go so long without knowing that? [18:32:45] i actually know another techy chadh [18:32:50] it's kinda confusing sometimes [18:32:55] when I did the conversion, I provided an authors file so that emails and user names were preserved in the commit log [18:33:16] what is chad's method? [18:34:05] ottomata: 21 13:20:26 < ^demon> hashar: Here's a start http://etherpad.wikimedia.org/Git-migration-tools [18:34:06] <^demon> Same thing, but I normalize everyone's e-mails to the fake @users.mediawiki.org (which may exist as a redirect one day) [18:34:26] <^demon> Also, the tool I uses retains the svn commit info as a ref/notes/* [18:34:38] ohhh, why the fakes? [18:34:41] o we don't want the emails [18:34:44] but the usernames are ok? [18:34:46] hmm [18:34:59] ^demon: so refs/notes/* is only old stuff not for new commits? [18:35:01] hmm, would it be better to let you convert this then? [18:35:09] i haven't made any changes to it since converting, we can start over [18:35:30] <^demon> jeremyb: refs/notes/* are a feature of git generally. The tool we use just stores the old commit info as refs/notes/commits [18:35:58] commit info == commit message? [18:37:19] what happens to stuff like eol-style? [18:37:37] is it recorded anywhere? [18:40:40] <^demon> ottomata: No, the commit messages are kept the same like they had been done in git initially. This just stores a link to the revision in code review, so you can know what SVN revision it referred to. [18:40:57] <^demon> jeremyb: eol-style commits aren't needed in git, and are discarded when converting. [18:41:13] ohhhh, cool [18:41:15] <^demon> Pretend they never happened ;-) [18:41:29] hmm, in that case should we redo that conversion? [18:41:50] <^demon> Yes we should :) I'm working on a set of rules to do a bunch of project conversions this week. [18:42:55] ok [18:43:03] rules as in a script? or explicit instructions? [18:43:34] ottomata: see the link i pasted [18:44:21] <^demon> We should triage /trunk/tools today :) [18:44:35] i see it [18:44:37] New patchset: Asher; "update package changelog" [operations/software] (master) - https://gerrit.wikimedia.org/r/8351 [18:44:39] not sure what that is [18:45:04] is that a script or just pseudo code? [18:45:13] New review: Asher; "(no comment)" [operations/software] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/8351 [18:45:15] Change merged: Asher; [operations/software] (master) - https://gerrit.wikimedia.org/r/8351 [18:45:26] <^demon> It's the configuration to map svn paths to git repos/branches on those repos. [18:45:31] oh [18:45:38] configuration to what? [18:46:00] <^demon> https://gerrit.wikimedia.org/r/gitweb?p=operations/software.git;a=tree;f=svn2git;hb=HEAD [18:46:04] <^demon> Magic tools :) [18:46:17] {{fact}} [18:47:18] hmm, oooook [18:47:23] <^demon> Chad says so [18:47:31] should I try to figure out how to use that, or should I let ^demon re-convert this repo for me? [18:47:41] :) [18:47:43] <^demon> You should let me do it with the others I'll be doing this week :) [18:47:47] ok perfect [18:47:59] i won't need to commit new changes until some other stuff gets deployed [18:48:00] so that's fine [18:48:15] I don't care about the name of the git repo, but it should be in the analytics/ project group [18:48:22] and you can delete the analytics/udplog one if you like [18:48:36] yes! [18:48:39] oops, wrong chat [18:48:45] <^demon> We can keep analytics/udplog, I'll just rewrite the history :) [18:50:58] ok cool [18:51:01] thanks! [18:56:27] ottomata: http://nagios.wikimedia.org/nagios/cgi-bin/status.cgi?hostgroup=analytics-eqiad [18:56:43] CooooooooOOOOl [18:56:54] eh actually, this one is better for bookmark: http://nagios.wikimedia.org/nagios/cgi-bin/status.cgi?hostgroup=analytics-eqiad&style=detail [18:57:21] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [19:03:35] mutante: as long as you're doing monitoring, search seems to be unmonitored in pmtpa? is that intentional? (works in eqiad i think) [19:03:39] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/6798 [19:03:41] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/6798 [19:04:04] * jeremyb asked about it a few times, just repeating because i saw you touching nagios [19:04:19] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/7751 [19:04:21] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7751 [19:09:43] jeremyb: stor [19:09:56] jeremyb: sorry, still in meeting, feel free to open RT [19:10:41] mutante: i think last time i tried to file an RT it turned me down (i don't exist) [19:11:03] jeremyb: email to ops-requests@ should work [19:11:16] @rt.wm.o ? [19:11:22] yea [19:15:23] looks like the last one i tried was to core-operations@ 1.5 yrs ago [19:15:36] * jeremyb will try again ;) [19:17:50] RobH: what would it take to get lucid working on the db61/62 hardware? [19:28:05] New review: Demon; "Wouldn't it make more sense to put this in bugzilla.pp?" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/8339 [19:31:13] ctwoo: can you forward me whatever you got from nagios this weekend re: rsearch? [19:33:02] notpeter: i just forwarded you the first page [19:33:11] it flapped 3 or 4 times [19:33:14] I didn't get a page either [19:33:21] i think it happened the week before as well [19:33:27] search-pool4 in all cases [19:33:59] huh, ok [19:34:01] thanks [19:35:35] notpeter: sometimes it's search1015, sometimes search1016 [19:35:59] notpeter: most is in SAL i think but IIRC not the most recent boot [19:36:58] (just guessing based on what asher/faidon's responses were ;) ) [19:37:16] heya [19:37:22] can someone review this with me today sometime? [19:37:22] https://gerrit.wikimedia.org/r/#/c/7988/ [19:39:04] jeremyb: gotcha [19:39:12] I mean, those are the two in that cluster [19:39:21] it might be time for more sharding... [19:39:47]