[00:01:05] RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Thu Jul 26 00:00:53 UTC 2012 [00:01:32] RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Thu Jul 26 00:01:29 UTC 2012 [00:02:35] RECOVERY - LDAPS on nfs2 is OK: TCP OK - 0.002 second response time on port 636 [00:03:11] RECOVERY - LDAP on nfs2 is OK: TCP OK - 0.000 second response time on port 389 [00:06:11] maplebed, RoanKattouw: I was looking at srv281 (it's on the server admin log) [00:06:18] notpeter: too [00:06:21] Yeah [00:06:24] it was disabled for quite some time due to disk full [00:06:27] It seems to have missed the repartitioning [00:06:37] I reformatted it, it was partioned with 7gb again [00:06:37] That's why its disk is still full [00:06:49] and now there's an additional problem [00:06:57] paravoid: I'm having an issue with lab [00:06:59] s [00:07:00] I reformatted it with our default distro, which is now precise [00:07:10] paravoid: I can't get to my mobile-testing instance [00:07:12] heh, we didn't fix partman, did we ? [00:07:19] which fails in some ways, so I have to fix that [00:07:22] paravoid: and I don't see anything on the console output page [00:07:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:29] New Apache installs are probably still partitioned wrong [00:07:34] I was looking at it but sidestepped it for something else [00:07:39] I think Peter just live-remounted the existing ones [00:07:46] preilly: is Ryan there? :) it's 3am here [00:07:52] paravoid: nope [00:07:58] paravoid: but don't worry about it I guess [00:07:59] okay, let me have a look [00:08:50] paravoid: Compare the output of 'mount' on srv281 and srv280 for instance [00:09:20] RoanKattouw: I'll have a look tomorrow, I just wanted to ping you because I saw you were wondering [00:09:29] you and maplebed [00:09:40] OK [00:09:53] it's disabled in pybal for quite some time [00:10:18] even if apache comes up, which I don't think it even can right now [00:12:36] paravoid: any idea why https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=mobile&instanceid=i-00000271 is blank? [00:12:53] none whatsoever :) [00:12:59] I'm trying to look at nova logs now [00:14:36] preilly: forgive me if it's a silly question: have you tried rebooting it? [00:14:51] paravoid: yes [00:16:05] PROBLEM - Host maerlant is DOWN: PING CRITICAL - Packet loss = 100% [00:18:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.723 seconds [00:19:02] that's strange [00:19:04] it's stuck in GRUB [00:20:07] preilly: seems to be up now [00:21:31] paravoid: you working right now or about to head out and enjoy the evening ? [00:21:48] LeslieCarr: "evening"? [00:21:51] "Evening" [00:21:54] 03:21 :) [00:21:57] (am) [00:21:59] hehe [00:22:07] s/evening/morning ? [00:22:10] paravoid: weird [00:23:17] preilly: mind if you pass it to Ryan or I investigate tomorrow? [00:23:35] paravoid: sure [00:23:37] the immediate problem should be fixed [00:23:41] i.e. the VM is up [00:23:49] let me find the root cause with a clear head :) [00:42:38] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [00:54:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:38] paravoid or anyone else who might know - i've made a bunch of changes in a labs instance with the self-hosted puppetmaster. i've made some local commits but now i want to push to gerrit [01:01:06] how do i do that with the clone in /var/lib/git/opperations/puppet? [01:03:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.146 seconds [01:15:33] sooo [01:15:38] why do so many memcached requests time out? [01:15:46] I guess need a decent testing for that [01:24:36] packet loss [01:24:54] that's my theory anyway [01:38:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 216 seconds [01:43:02] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 261 seconds [01:49:47] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 668s [01:49:56] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.766 seconds [01:52:38] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 17 seconds [01:55:29] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 8s [01:55:56] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 11 seconds [02:22:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:31] New patchset: preilly; "switch back to strtok instead of strtok_r" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16716 [02:25:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16716 [02:25:22] New patchset: preilly; "remove carrier acl block and switch back to strtok instead of strtok_r" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16716 [02:26:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16716 [02:27:31] any operations people actually here right now? [02:30:28] mark: ping [02:30:40] paravoid: ping [02:30:45] notpeter: ping [02:34:02] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.686 seconds [02:44:31] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Thu Jul 26 02:44:05 UTC 2012 [03:11:58] RECOVERY - Puppet freshness on srv198 is OK: puppet ran at Thu Jul 26 03:11:48 UTC 2012 [04:04:25] gerrit acct creation, already has SVN (so I can't do it myself): https://www.mediawiki.org/w/index.php?title=Developer_access&diff=565442&oldid=565340 [04:14:28] * jeremyb waves Ryan_Lane [04:15:23] preilly was having a labs problem. paravoid poked it briefly and got the instance up but didn't really investigate because it was 3am. then preilly came back again looking for ops later but didn't say why and then he /quit [04:16:44] instance was 'mobile-testing' [04:26:54] anyone around? [04:39:52] preilly: mobile-testing still? [04:40:17] jeremyb: nope [04:50:08] PROBLEM - Puppet freshness on potassium is CRITICAL: Puppet has not run in the last 10 hours [05:56:51] anyone around? [07:09:52] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [07:33:52] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [08:04:37] hello hashar [08:04:45] good morning :-) [08:05:37] hashar: is there a quick query you can make to see how many reviews (excluding bot reviews) we're having on gerrit compared to CodeReview? [08:05:59] Nemo_bis: I don't have access to gerrit query engine :-D [08:06:07] uh [08:06:14] Nemo_bis: I know the analytics team is polishing a tool that will generate statistics out of Gerrit [08:06:18] who does, only demon? [08:06:24] like the number of changes merged per day and per repo [08:06:32] the time between patch submission and its +2 [08:06:34] yep, I hoped for a quick and dirty answer [08:07:26] there are some experimental data in analytics/gerrit-stats/data.git (though you might not have access) [08:07:54] git clone ssh://gerrit.wikimedia.org:29418/analytics/gerrit-stats/data.git [08:07:54] :) [08:08:20] then you get a datafiles/mediawiki/core/core.csv file [08:08:24] might get what you want [08:26:43] hashar: I do have permissions [08:27:18] great!!! [08:27:50] and also sent wikitech email as a plus [08:33:58] Change merged: Nikerabbit; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16498 [08:35:01] PROBLEM - Puppet freshness on cp1020 is CRITICAL: Puppet has not run in the last 10 hours [08:36:01] hashar: contains the following columns, not so useful IMHO: date,commits,self_review,time_first_review_staff,time_first_review_total,time_first_review_volunteer,time_plus2_staff,time_plus2_total,time_plus2_volunteer [08:36:04] PROBLEM - Puppet freshness on mw58 is CRITICAL: Puppet has not run in the last 10 hours [08:36:13] Nemo_bis: sorry that is all I got for now :-D [08:36:20] hashar: yeah [08:36:41] also probably broken, or it would mean we have no reviews at all or so [08:36:52] so we have to wait a bit more :) [08:36:55] Nemo_bis: you can talk about it with Diederik van Liere :-D [08:36:58] he wrote the code [08:37:31] I think the Gerrit report card will be released in the next few weeks. They will be able to tell you [08:37:50] he said "next week" 23 days ago :) [08:38:00] ask him again so :-] [08:38:02] PROBLEM - Puppet freshness on db35 is CRITICAL: Puppet has not run in the last 10 hours [08:38:02] PROBLEM - Puppet freshness on srv209 is CRITICAL: Puppet has not run in the last 10 hours [08:38:05] though he is sleeping right now hehe [08:38:08] but yes it's difficult [08:38:24] I don't think he needs a reminder :) [08:38:53] at least he can gives you an updated deadline :) [08:39:23] oh well, I think a talk comment and a wikitech email is enough [09:44:11] morning [09:46:23] hello paravoid :-] Had a good night? [09:46:44] * hashar looks at Athens weather http://www.bbc.co.uk/weather/264371 [09:47:24] ouch low of 30°C during night, which is what we got during the afternoon here and that is basically rendering everyone useless :-/ [09:48:13] paravoid: I got a fresh easy hack for you to review https://gerrit.wikimedia.org/r/#/c/16661/ :-D [09:48:28] then we can do the NFS to /data/project migration if you feel like doing it on wake up [10:13:21] New patchset: Ori.livneh; "*UNTESTED* Calls to bits-lb.eqiad/event.gif 204'd" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16724 [10:13:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16724 [10:26:47] mark: around? [10:27:02] mark: 1: cs:Connected ro:Secondary/Secondary ds:UpToDate/Diskless A r---- [10:27:17] mark: diskless means broken I think :) [10:32:25] hashar: it's a horrible *horrible* hack [10:32:30] 16661 that is [10:35:33] paravoid: not nastier than the existing one :-D [10:36:25] maybe that could be set at the realm.pp level, but I am not really willing to clean that out [10:43:59] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [10:55:51] paravoid: yeah [10:55:54] not saying that isn't broken [10:55:59] just saying, it's normal that isn't mounted [10:56:12] ok [11:00:54] http://www.theregister.co.uk/2008/07/18/hp_packaging/ [11:00:55] this is so true [11:00:57] I hate HP [11:01:12] once we got like 2 pallets of HP boxes for toolserver [11:01:16] Dell is pretty good for packaging [11:25:53] hahaha that's so very true [11:37:28] New review: Alex Monk; "Almost perfect, just a couple of nitpicks." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/16035 [12:25:33] I'm seriously thinking of sending all labs mail to /dev/null until we have a proper relay [12:25:45] we're essentially blackholing them in our mailboxes anyway [12:32:22] ah you enjoyed the fcron mails? [12:33:13] yes [12:34:09] hehe [12:55:42] hashar: nfs? [12:56:00] paravoid: sure :-) [13:00:04] mark: if you are in a review mood, I have updated/rebased my two patches for bits.beta.wmflabs.org https://gerrit.wikimedia.org/r/#/c/15445 (fix bits when enable_geoiplookup is enabled) https://gerrit.wikimedia.org/r/#/c/13304/ (use the new cluster_options hash to pass the test hostname). [13:00:29] mark: oh and it is deployed on labs via puppetmaster:self :) [13:00:59] paravoid: ready for it ? change is https://gerrit.wikimedia.org/r/#/c/15545/ [13:05:29] New review: Faidon; "finally :)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/15545 [13:05:34] what could possibly go wrong [13:06:34] ah, I have to do 16632 first [13:06:49] yup that one could be nasty [13:06:55] but I think it fix an issue we have currently [13:07:38] some servers not having lvsrealserver end up with no nfs::upload [13:07:50] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16632 [13:07:54] hasher, Nemo_bis: are you around? [13:07:56] though I guess puppet is not smart enough to actually magically umount upload :D [13:08:01] drdee: I am there [13:08:19] yes, things have been slow with gerrit-stats, main reason was the work that needed to be done on limn [13:08:35] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15545 [13:08:40] now that's out of the door so to say, it's on github, we are now focusing on gerrit-stats [13:08:57] drdee_: I am fine with it :-D Maybe you could write a short message on wikitech-l so volunteer know about it ? [13:09:13] we deployed a first version on labs, but there are some issues i need to iron out first [13:09:16] drdee_: I am myself 100% confident that project will be successfull and don't really care when it will land :-D [13:09:42] paravoid: running puppet on the instances [13:12:05] paravoid: the f**** stupid puppet is file bucking everything :-D [13:12:16] info: /Stage[main]/Nfs::Apache::Labs/File[/usr/local/apache]: Filebucketed /usr/local/apache/common-back/php-1.19/languages/messages/.svn/text-base/MessagesPrg.php.svn-base to puppet with sum 3b2f65a59d80da76a8ce3220f339c6bd [13:13:04] uh oh [13:13:17] notice: Finished catalog run in 115.29 seconds [13:13:19] anyway :) [13:13:35] there's a way to tell it to not do that [13:13:40] don't remember the details [13:13:43] anyway, did it work? [13:13:48] not sure [13:14:01] need to manually umount the old point [13:14:02] I didn't push it to production yet [13:16:03] I have manually umounted /mnt/upload6 and rerunning puppet [13:16:58] hello drdee_ [13:17:10] hey Nemo_bis [13:17:25] oh focus on gerrit-stats, nice [13:17:29] yes we kept you waiting :) [13:17:31] or :( [13:17:35] haha [13:17:37] blaahhh info: /Stage[main]/Apaches::Service/Exec[apache-trigger-mw-sync]: Scheduling refresh of Exec[mw-sync] [13:17:42] never going to work on lab I guess [13:18:00] the work on limn was taking more time: github.com/wikimedia/limn [13:18:14] but that's done (for the moment) [13:18:19] drdee_: are you including per-reviewer stats for code review? (to answer questions like the last one I sent to wikitech-l)? [13:18:35] not in the initial version, but i would like to add that [13:18:51] because now with gated trunk we need social pressure for code review or it will never get done I think [13:18:51] what per reviewer stats are you thinking about? [13:19:06] well, who reviews how much for instance [13:19:33] maybe even how responsive people are to review requests (bt also how many requests do they receive) [13:19:43] paravoid: lets move to labs :) [13:20:38] drdee_: I suppose you're alredy going to make nice graphs of the number of unreviewed commits and such of course [13:20:55] ok, i'll note those per reviewer stats [13:21:05] yes, everything will be visualized using limn [13:21:38] the "etc." on (3) in http://lists.wikimedia.org/pipermail/wikitech-l/2012-April/060248.html is quite broad :) [13:21:59] ohhh about the DOI URI support [13:22:31] uh you were in cc I guess [13:22:37] you know that DOI standard is crazy? and that there are many more DOI formats then just doi:10.1000/186 ? [13:22:52] sigh [13:22:54] yeah, in berlin i was pushing for this as well [13:23:01] it's actually not easy [13:23:12] because basically any character is allowed in a DOI [13:23:18] someone mentioned in the bug that the other forms are nonstandard? [13:23:25] Logged the message, Master [13:25:09] drdee_: also things like number of reviews (not only of -1/-2/+1/+2), maybe even inline comments [13:25:23] * Nemo_bis just thinking out loud [13:26:51] so what i have now is: number of commits per day, number of days until first review (excluding bot reviews), number of days until +2, and this is a breakdown by staff and volunteers [13:27:24] but i am sure those metrics need to be refined, and we wil want to add more [13:28:53] nemo_bis: http://www.doi.org/doi_handbook/2_Numbering.html#2.5 and read the first line [13:30:37] New patchset: Nikerabbit; "Initial version of solr for ttmserver" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16732 [13:30:51] drdee_: those are general metrics on how well we're doing to verify there aren't too many forgotten commits [13:31:09] but not really a way to measure how much code review activity we have [13:31:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16732 [13:31:17] yes, but that is , AFAIK, the biggest concern right now [13:31:30] adding new metrics is quite straightforward [13:31:30] and even less a way to show who is doing such activity [13:31:59] oh and there is a metric for the percentage of self-review [13:32:00] I know that's the concern, but as I said we probably have a need for social pressure towards code review [13:32:29] haha we need a leader board :) and hand out badgges [13:34:26] drdee_: also, I assume basic stuff like number of commits per committer (besides reviews and merges) are in the plan? [13:34:43] drdee_: is there a mediawiki.org page or something for these specifications? [13:34:53] well the unit of analysis is the repo, not an individual [13:35:19] drdee_: but Erik mentioned also the individual as possible target, although with lower priority [13:35:32] yep, lower priority [13:35:56] but i am not sure that i like this carrot/ stick approach [13:35:57] drdee_: which still needs to be tracked somewhere so that some day someone does it :) [13:36:21] drdee_: carrot/stick is "how many -1 you have received, bad boy?" [13:36:46] just the number of reviews/merge/commits is normal stats [13:37:05] (basic) [13:37:19] so number of commits is present [13:37:32] Nemo_bis: about the gate trunk, the changes that are reviewed are now deployed in the next 2 weeks. So that is at least an improvement :-] [13:37:49] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [13:37:56] number of reviews is not necessarily useful, you have multiple patch sets per commit, you can have multiple reviewers per commit [13:38:31] the focus right now is how long do you have to wait for feedback [13:38:35] hashar: of course :) [13:38:49] and how long does it take to get your commit merged [13:38:59] and is there a difference between volunteers and staff [13:39:13] and this is on a per repo basis [13:39:16] drdee_: sure sure, I only want to understand if you think it makes sense to have stats on individual contributions too, at some point [13:39:25] platform eng does a lot of pair to pair review so we probably have a lower latency. [13:39:46] say I exchange 3 reviews from aaron vs me reviewing two of is changes. [13:39:46] hashar: uh, like citation market in scientific publishing? :D [13:40:04] we can count how often a dev does a review but that needs more context to be actionable [13:40:13] like we are closely collaborating and are available everyday [13:40:33] :p [13:40:34] whereas with a volunteer you get some inherent latency cause volunteers are not there everyday [13:40:42] and of course staff don't do that much review during the week-end :-] [13:40:58] drdee_: yes, stats don't need to be directly actionable do they [13:41:02] that is why I do my review on monday, I can apply the volunteer work that has been done while I am enjoying some fresh air and my family :-]]]]]]] [13:41:11] so yeah we will get a difference for sure [13:41:32] well if stats are not actionable, what is the purpose? [13:42:51] well if stat mean time for merge is 3 days and volunteers is 5, I don't think we have a problem [13:43:04] if staff is 2days and volunteers is 25 days that is entirely different :-] [13:43:04] drdee_: I mean that they can be neutral, don't suggest any action per se [13:43:16] stats are never neutral :D [13:43:28] well they can be more or less so [13:43:42] drdee_: is there any place we can open feature requests ? [13:43:58] one interesting thing would be an aggregate of extensions deployed on wmf vs non wmf extensions [13:44:06] is already done [13:44:18] drdee_: do you think that a "rank" of committers by number of commits/merges/reviews is too biased/wrong in some way [13:44:20] but yes, we need a place to file feature requests [13:44:25] !log deployment-prep rsync finished for both apache and upload6. Remounting and restarting apaches [13:44:32] Logged the message, Master [13:45:08] hashar: wrong channel? [13:45:09] :) [13:45:12] Nemo_bis i wouldn't want to go into that direction personally, but i am open for debate [13:45:50] grmblblblbl [13:45:57] I need to set different background colors [13:45:59] :) [13:47:18] drdee_: ok where should the debate happen? :) [13:47:40] where is the direction we're going to tracked? [13:47:59] probably as a follow up to the initial announcement of gerrit-stats [13:48:08] drdee_: I think the best thing is just to allow people to get their stats by themselves, which AFAICS is the way you're going [13:48:13] ok [13:48:17] not sure if i understand your question [13:48:43] so, code review DB was replicated to toolserver and people extracted stats there [13:49:22] drdee_: which question? I mean if a list like http://lists.wikimedia.org/pipermail/wikitech-l/2012-April/060248.html has been placed somewhere on wiki with subsequent updates etc. [13:49:34] mark: srv/mw partioning; needs fixing; while at it, do you want me to make any larger changes? [13:49:44] mark: like replace jfs or…? [13:50:43] yeah replace jfs [13:50:58] but we no longer need /a at all [13:51:09] perhaps make a larger /, and keep a lot of free space [13:51:16] we can always add partitions or LVM later if we suddenly need to [13:51:36] so put everything into a large / ? [13:51:36] image/video scalers can then use that space [13:51:47] depends on what you mean by 'everything' [13:51:57] and /a? not /tmp? [13:52:11] oh is that what peter renamed it to? [13:52:12] no idea [13:52:23] yeah apaches need a /tmp space, perhaps make that a special partition [13:52:45] i wonder if /usr/local should be separate, where mediawiki lives [13:53:13] curerently there's a 7gb / (that's filled up), a 65g /a and a 2g /tmp [13:53:24] uh ok [13:53:30] ext3 jfs ext3 respectively [13:53:34] make / 30 GB or thereabouts [13:53:48] 30g / ext3, 2g /tmp ext3 sounds good? [13:53:48] make a decent sized /tmp, 10 GB or so [13:53:52] 10? [13:54:05] if we want image scalers to use that yes [13:54:07] well, I guess we have 250g disks there [13:54:11] yeah [13:54:23] the least you would possibly find is 80 GB [13:54:26] but I think we don't have those [13:54:48] okay then [13:54:52] anyway, keep the remaining space unpartitioned or unused in LVM I think [13:54:54] then we have flexibility [13:55:04] okay! [13:56:37] hm, interesting, srv280 has a separate /usr/local/apache and no /a [13:56:44] but that's not in autoinstall [14:02:10] hashar: do you know what the current process is for apache config changes? I recall something about them being in both git and svn. [14:03:53] maplebed: I think we have almost everything in git now [14:04:03] Tim did cleanup them [14:04:35] and do you know what the deploy process is then? the wiki still says to run sync on fenari without any mention of git. [14:04:58] maplebed: yeah the local svn got archived [14:05:07] in /h/w/conf/httpd/archive [14:05:14] so it is 100% git / gerrit [14:05:15] oh wait, I was looking at the wrong page. [14:05:26] which page were you looking at ? [14:05:30] http://wikitech.wikimedia.org/view/Sync_scripts#Operating_on_apaches_and_image_scalers_dsh_groups [14:05:53] http://wikitech.wikimedia.org/view/Apaches#Deploying_config does say do it in gerrit first. [14:06:08] [[Apaches]] should be the reference now [14:06:14] I think I got proofread by mutante [14:06:35] maplebed: we're supposed to talk about an RT ticket :) [14:06:43] so we are. [14:06:52] maplebed: good morning btw, you're an early riser! [14:06:56] though ^^^ is relevant too. [14:07:04] (or a party animal) [14:07:09] so anyway [14:07:12] since part of the thing in between me and doing that ticket was figuring out how to deploy apache configs. [14:07:13] :P [14:07:21] there might be files that are ignored by git [14:07:22] paravoid: sadly not so much. It's already past 10am. [14:07:22] yeah, that's how I remembered it [14:07:23] havent checked [14:07:35] hmm no there is not :-) [14:07:36] maplebed: oh? not in SF? [14:07:43] we are 100% in git!!! yeah [14:07:57] thanks hashar. I think you've gotten me far enough that I'll be able to stage the change I want to make. [14:07:59] :) [14:08:09] so basically [14:08:26] copy the repo locally, submit change, have it reviewed/merged, git pull on fenari, sync-apache [14:08:27] paravoid: on cape cod (MA) [14:08:36] ohhh sneak the test suite before running sync-apache [14:08:47] can't remember where it is though but Jeff Green wrote a mail about is perl script [14:10:16] paravoid: I got as far as confirming that the rewrite rule should just send everything to index.php and that it should go in main.conf. [14:10:34] I didn't get to test it and just now got to figure out how to stage / review it. [14:10:43] I think that's actually about it. [14:11:02] (the hardest part was sorting through the tickets to find out what folks actually wanted) [14:11:45] if you find bugzilla easier, maybe you could have the apache-config request set there instead of RT ? [14:11:54] + you will get support from the volunteers :-) [14:12:02] (re: index.php - the rewrite rule is not supposed to send traffice to Special:ShortURL) [14:12:28] hashar: the problem was that there were 2 or 3 bugzilla tickets and an RT ticket, all with different versions of the request, and within each ticket, each request morphed through the comments. [14:13:18] having two issue tracker surely does not help [14:13:31] maybe those non private requests can be full filed via bugzilla [14:13:41] but then ops might not want to track bugs in both bugzilla and RT [14:13:59] what I mean: the biggest problem wasn't the choice of ticket tracking system [14:14:20] sure, it didn't help. but it wasn't the source of confusion. [14:15:35] paravoid: can we get the syslog hack in please? https://gerrit.wikimedia.org/r/#/c/16661/ [14:26:25] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [14:29:34] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [14:50:44] PROBLEM - Puppet freshness on potassium is CRITICAL: Puppet has not run in the last 10 hours [14:53:49] 26 04:04:24 < jeremyb> gerrit acct creation, already has SVN (so I can't do it myself): https://www.mediawiki.org/w/index.php?title=Developer_access&diff=565442&oldid=565340 [14:55:31] New patchset: Mark Bergsma; "cp1041 (Precise) mobile disk cache back to 100G" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16739 [14:56:13] New patchset: Mark Bergsma; "Add asw-c-eqiad to Torrus" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16641 [14:56:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16641 [14:56:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/16739 [14:56:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/16739 [14:58:39] maplebed: so, I presume you don't know more about that short url thing [14:58:45] and I should ask the requesters [14:58:56] mark: how's the varnish stuff? working so far? [14:59:12] yes [14:59:15] cool [14:59:16] the persistent storage is [14:59:19] right [14:59:20] streaming hasn't been tested yet [14:59:28] right, yeah, I got that [14:59:53]