[00:06:26] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [00:37:48] New patchset: preilly; "add dolphin browser assets to bits docroot" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11973 [00:38:00] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11973 [00:40:31] New review: Tim Starling; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11973 [00:40:35] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11973 [01:08:32] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [01:41:32] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 260 seconds [01:43:20] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 279 seconds [01:47:23] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 30 seconds [01:49:29] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 648s [01:50:59] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 37s [01:52:02] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 2 seconds [02:06:08] PROBLEM - Host mw1148 is DOWN: PING CRITICAL - Packet loss = 100% [02:28:29] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [03:01:29] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Tue Jun 19 03:00:57 UTC 2012 [04:05:32] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [04:56:32] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [07:14:33] !log reboot snapshot2, package and kernel updates [07:14:39] Logged the message, Master [07:15:40] PROBLEM - Host snapshot2 is DOWN: PING CRITICAL - Packet loss = 100% [07:17:55] RECOVERY - Host snapshot2 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [07:27:43] !log reboot snapshot1, package and kernel updates [07:27:48] Logged the message, Master [08:06:09] New patchset: Dereckson; "(bug 37699) Change logo on uz.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11977 [08:06:16] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11977 [08:17:36] New review: Dereckson; "Waiting local consensus URL (shellpolicy)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/11977 [08:24:00] New patchset: Catrope; "Move pgehres from restricted to mortals" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11979 [08:24:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/11979 [08:27:27] New review: Dereckson; "The configuration is fine." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/11943 [08:55:57] New review: Hashar; "''rephrasing my previous comment''" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/9130 [09:02:11] New review: Nikerabbit; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/11833 [09:02:13] Change merged: Nikerabbit; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11833 [09:03:29] New patchset: ArielGlenn; "run as root; ConnectTimeout opt; correct basedir check" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/11981 [09:04:28] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11981 [09:04:30] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/11981 [09:26:11] New patchset: Petrb; "(Bug 37700) - Change logo for stewardwiki to http://commons.wikimedia.org/wiki/File:Steward_wiki_logo_3.svg and favicon to meta one." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11943 [09:26:18] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11943 [09:31:21] New review: Dereckson; "Please read my review comments in the previous changeset, as they are also valid for your changeset ..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/11943 [09:36:58] New review: Dereckson; "(no comment)" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/11150 [09:41:17] !log updating TranslationNotifications extension with NikeRabbit [09:41:22] Logged the message, Master [09:59:55] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [10:05:31] !log TranslationNotifications extension updated by Nikerabbit! [10:05:36] Logged the message, Master [10:07:52] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [10:27:59] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/11943 [10:30:52] paravoid: so, everything is switched over [10:31:04] paravoid: I'm going to give it a little while before I remove the 135 address [10:44:45] New patchset: Ryan Lane; "Add second master for labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11986 [10:45:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/11986 [10:45:22] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/11986 [10:45:24] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11986 [11:03:13] !log adding IPs for virt6-8 [11:03:18] Logged the message, Master [11:09:49] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [11:15:09] Ryan_Lane: great [11:29:46] New patchset: Petrb; "(Bug 37700) - Change logo for stewardwiki to http://commons.wikimedia.org/wiki/File:Steward_wiki_logo_3.svg and favicon to meta one." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11943 [11:29:52] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11943 [11:31:28] New patchset: Petrb; "(Bug 37700) - Change logo for stewardwiki to http://commons.wikimedia.org/wiki/File:Steward_wiki_logo_3.svg and favicon to meta one." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11943 [11:31:34] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11943 [11:31:54] New review: Petrb; "I changed the format everywhere, if there is a standard for these comments it should be effective fo..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11943 [11:50:45] Ryan_Lane: did you see the https thread on wikitech? [11:51:35] mark: ^ too [11:51:52] mozilla contcted us; they want to switch the wikipedia search box to https by default [12:00:49] paravoid: which https thread? [12:00:50] oh [12:00:51] that one [12:01:01] so, I doubt we'll see much traffic from it [12:01:08] but it sends a bad precedent. [12:01:13] *sets [12:01:33] we aren't doing https for anons. If we change search, it's sending anons to https. [12:01:50] I'd have less issue if we had https on every server [12:03:09] we're currently getting about 20MB/s across all of the nodes in all datacenters for https [12:03:48] the servers are pretty bored, but that's not much traffic [12:03:49] New review: Hashar; "Just forget about the bug format in comment, it clutter the 'git blame' history making it hard to fi..." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/11943 [12:04:21] if we send 30x the traffic to them we'll saturate their links [12:04:53] well, that's esams anyay [12:04:59] I guess more like 50x total [12:05:40] that's with no hardware dead, and not needing to failover between sites. so, realistically, 20x more traffic. [12:05:46] * Ryan_Lane shrugs [12:06:59] New review: Mark Bergsma; "Almost there. :)" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/11574 [12:07:38] what do you mean by "we're not doing https for anons"? [12:10:46] mark: meaning, we're not sending anons to https [12:11:12] if they manually go to https, then they hit it, but otherwise they don't [12:11:43] I'd really like to see the load for https with logged in users using it by default [12:11:59] then decide if it would be a huge hit to put it on all squid/varnish frontends [12:17:06] mark: btw, I'm nearly positive you have the most gerrit commits out of everyone [12:17:11] you have over 1000 [12:17:49] hehe [12:17:56] not so many recently [12:18:02] it kind of depends on what i'm working on, that varies ;) [12:18:31] you cheat, though [12:18:56] you push entire branches in as separate changes [12:21:17] no, you're an idiot for not doing that ;) [12:21:21] :D [12:21:31] I do feature branches [12:21:38] so do I [12:21:41] but I dislike squashing [12:21:48] think it defeats the point of git [12:21:57] * Ryan_Lane nods [12:22:29] the change itself has history, if you do patchsets [12:26:29] I agree wholeheartedly on that [12:28:38] well, you can do a merge commit, but then you lose review capability, which sucks [12:29:06] I wish the merge commits worked like normal commits, and you could review what the merge would be [12:29:18] if you need to review an entire branch in gerrit as one commit, you've already lost [12:29:30] that's how github works [12:29:36] no [12:29:47] you review the merge as a single pull request [12:30:01] yes, but you keep the individual commits [12:30:13] hello [12:30:18] hashar: hi :) [12:30:22] Ryan_Lane: can we submit merges in gerrit ? [12:30:24] but technically, you review the entire branch as one commit ;) [12:30:32] hashar: yes, but then you can't code review [12:30:47] if just that one thing was fixed in gerrit, I'm sure it would make a ton of people happy [12:31:12] I wanted to create a fork of operations/production , cherry-pick several changes from tests then submit that branch for merge in production [12:31:16] of course it needs to be a no-ff merge to work [12:31:35] Ryan_Lane: no. you review the *pull*, but you can review individual commits and you keep those when you merge [12:31:37] I was thinking of spending today merging the branches [12:31:42] yeahhh [12:32:08] I talked about merging test /production a while back with Daniel Zahn [12:32:16] it's not an easy task [12:32:19] Ryan_Lane: also, I'm not sure if it's gerrit's or our fault, but the way our puppet git repo is, is driving me crazy [12:32:24] one way would have been to have each person to cherry pick its changes [12:32:26] every second commit is a merge [12:32:27] paravoid: what do you mean? [12:32:28] starting from the older ones [12:32:45] paravoid: that's because we're cherry-picking [12:32:48] paravoid: git log --no-merges [12:33:03] we are cherry-picking and allow gerrit to merge changes [12:33:16] I wonder if I have always-merge set in our repo [12:33:20] lemme check [12:33:25] I have set my integration repo to be ff only [12:33:28] I don't understand why cherry-picking results in merges. [12:33:42] nope. "Merge-if-necessary" [12:33:45] cause the parent of the commit is not the latest master [12:33:55] indeed. so it needs to merge [12:34:11] why can't if fast forward? [12:34:12] if I changed to "Fast-forward-only" we wouldn't have that [12:34:15] the only thing that is puzzling me is why it does not rebase first [12:34:19] right [12:34:20] but we'd have to rebase way more often [12:34:32] with Gerrit 2.4 (I think) we will have a [Rebase] button :-] [12:34:43] if we used git-review it would rebase for us [12:34:54] https://gerrit.wikimedia.org/r/#/admin/projects/integration/jenkins <-- fast forward only :-]]° [12:34:56] I think most ops people aren't using git-review [12:35:31] I guess most ops don't care about review [12:35:32] I don't understand why gerrit is not rebasing it instead of us doing it on the client [12:35:50] no clue [12:36:12] maybe it thinks that would be a bad assumption to make for the client? [12:36:17] hashar: speaking just for myself, I find review is important but we're using puppet for doing pretty much everything and you can't expect two people to sign off every little change you make [12:36:28] yep [12:36:30] it's like reviewing my shell [12:36:41] we need that for my DNS actions [12:36:44] oh I don't use shell any more [12:36:51] exactly [12:36:57] too cumbersome to wait for whoever to validate my command when I ever I press enter [12:37:03] I think reviews totally make sense for larger changes [12:37:13] I was just ranting anyway [12:37:13] yes. we ask for review on those [12:37:32] I often ask for reviews when people are around, too [12:37:50] I don't care about ops self merging their change. Ops know what they are doing and if they screw something they are responsible for it :) [12:37:55] we really need git annotations [12:37:56] we somehow do the same for mediawiki/core [12:38:02] git notes? what the hell are they called? [12:38:10] done via gerrit, so we can do review after merge [12:38:18] http://www.kernel.org/pub/software/scm/git/docs/git-notes.html [12:38:21] it's on gerrit's roadmap [12:38:39] OpenStack people do export notes from Gerrit to their Git repo I think [12:38:44] they do [12:38:50] this came up at the last design summit [12:38:52] I suggested it [12:39:11] I usually just git show , then copy paste the change id in Gerrit interface [12:40:01] paravoid: speaking of review, I have a debian package change for you to review one day ;-) https://gerrit.wikimedia.org/r/#/c/11610/ [12:40:40] New patchset: Ryan Lane; "Add dhcp entries for virt6-8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11990 [12:40:51] Ryan_Lane: on the topic of production/test merging, m.ark did an attempt and reverted it. So the history is a bit screwed ;--D [12:40:54] paravoid: I'm not sure if Leslie did the networking for the new virt hosts [12:41:10] paravoid: we may want to tell them to hold off on everything except for virt6-8 [12:41:16] since we want to replace 1-4 [12:41:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/11990 [12:41:44] hashar: yeah. it's going to take me all day to successfully do the merge [12:41:57] Ryan_Lane: ? [12:42:12] Ryan_Lane: also, wasn't I supposed to do virt6-8 setup? :) [12:42:16] Ryan_Lane: I believe that should be some kind of a team job. [12:42:17] paravoid: well, right now they are set up at virt9-12 or something like that [12:42:25] paravoid: yeah. I just did dns and dhcp [12:42:28] I am pretty sure the ganglia stuff can be easily merged by Sara as an exemple [12:43:15] hashar: how can I see a cherry-pick difference between them? [12:43:38] what are the reasons again why we can't have a branch per labs project? [12:43:44] because of course this will happen again [12:43:54] there is: git cherry -v production test [12:44:01] New review: Faidon; "LGTM. I think there's a spurious start on init's gracefula action but this was before this change an..." [operations/debs/wikimedia-job-runner] (master); V: 0 C: 1; - https://gerrit.wikimedia.org/r/11610 [12:44:44] also git log production...test [12:44:50] I am not sure what the triple dot does though [12:45:19] seems to be a list of all changes in production and test based on a common ancestor [12:46:19] I'll probably just do a git merge production, from test [12:46:21] and figure it out [12:46:30] then push in a merge commit [12:47:05] then merge the opposite way and do it again [12:47:23] then you got to figure out all the planet, openswift, ganglia stuff (and more ) http://dpaste.org/Df4wo/ [12:47:35] that's fine [12:47:40] most of that I did anyway [12:47:43] just for ganglia, you have legit changes in bots prod and test :-( [12:47:50] bots -> boths [12:47:56] well, not the swift stuff [12:48:27] sometime I feel each project could get its own puppet repo ;-D [12:48:29] going from test to production is going to be much harder [12:48:34] just like mediawiki extensions [12:48:43] not that we have self:puppet, I want to get rid of the test branch [12:48:48] and run labs from production [12:48:58] yesss [12:49:03] ;) [12:49:09] that's why I was going to take the time to merge the branches [12:49:20] just throw test away ;-) [12:49:21] * mark ducks [12:49:27] can't do that :) [12:49:38] it would break labs horribly [12:49:45] that's what labs is for [12:49:48] I've tried to always cherry-pick across needed changes [12:49:54] mark: I can't have constant outages ;) [12:50:03] true you can't [12:50:06] but I still have credit! ;-) [12:50:10] hahahaha [12:52:21] New patchset: Hashar; "(bug 37644) Enable subpages on be.wikimedia.org for NS 0 and 4" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11723 [12:52:34] I am going to break the cluster configuration this afternoon [12:52:42] I am proceeding mediawiki-config queue [12:53:07] wtf gerrit? [12:53:09] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11723 [12:53:19] it's taking forever to merge my change :( [12:53:25] we really need dbs in eqiad [12:53:25] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/11990 [12:53:26] just took some long time too [12:53:45] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11723 [12:53:52] or move Gerrit to pmtpa ? ;-] [12:53:52] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11990 [12:53:56] * Ryan_Lane sighs [12:54:04] I really don't want to move the gerrit server [12:54:15] it's a pain in the ass [12:54:17] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11723 [12:54:26] so move the DB! ;-D [12:54:35] we don't have misc db servers in eqiad [12:54:42] I have a procurement rt in [12:55:06] is that because you are lacking the hardware to do so ? [12:55:06] I think delta-compression is making gerrit slower for reviews and merges and such [12:55:15] yes. there's no misc db hardware in eqiad [12:55:25] hence having a procurement ticket in ;) [12:55:35] I have no idea what a procurement ticket is :( [12:55:50] ah. heh [12:55:57] is that something like "pliizzz gimme $$$ 4 my project?" [12:56:09] no. it's "please buy this hardware" [12:58:09] New patchset: Hashar; "(bug 37661) Change Vietnamese Wikibooks logo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11747 [12:58:23] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11747 [12:58:39] New patchset: Ryan Lane; "Fix dhcp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11992 [12:59:06] Ryan_Lane: anyway there must be another issue on manganese [12:59:08] New review: Hashar; "Logo protected http://vi.wikibooks.org/wiki/file:Wiki.png?uselang=en so we are safe to go :-)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11747 [12:59:08] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11747 [12:59:09] ok. what's going on with the gerrit server? [12:59:09] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11992 [12:59:09] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/11992 [13:00:09] nothing for the last 4 hours on manganese http://ganglia.wikimedia.org/latest/?r=4hr&cs=&ce=&m=&c=Miscellaneous+eqiad&h=manganese.wikimedia.org&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [13:00:19] I'm wondering if its the dbs [13:00:33] 1 st blame the network and DNS [13:00:41] 2nd blame the intern software developer [13:00:45] 3rd db :-] [13:04:13] New review: Hashar; "The logo at http://commons.wikimedia.org/wiki/File:Wikipedia-logo-chr.png is not protected against e..." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/11741 [13:04:39] Ryan_Lane: bugzilla is slow too [13:04:45] so might be db9 [13:04:55] or whatever db is for misc [13:08:54] New review: Hashar; "I am afraid this is going to allow import from any french or italian project." [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/11746 [13:09:44] yeah. is likely db9 [13:10:20] I don't see any long running queries [13:10:36] mysql seems quite busy, though [13:11:27] observium seems to be doing schema updates [13:11:43] for its eventlog [13:13:25] New patchset: Hashar; "(bug 37457) viwikibooks can import from fr/it wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11746 [13:13:32] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11746 [13:14:33] New review: Hashar; "Patchset 3:" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/11746 [13:17:59] New review: Hashar; "Please rewrite your commit message to something shorter :) See: http://www.mediawiki.org/wiki/Git/Co..." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/11748 [13:19:27] well, so far production -> test isn't horrible [13:19:41] New review: Hashar; "The file at http://commons.wikimedia.org/wiki/File:Wikipedia-logo-v2-uz.png needs to be protected fi..." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/11977 [13:20:17] I have no clue about the swift stuff, though [13:20:37] you will want to merge that with Ben assistance I guess [13:20:41] cmjohnson1: sure. lemme shut it down [13:21:09] well, I think the stuff in test is newer [13:21:29] it is [13:21:46] cmjohnson1: should be down [13:23:04] so here is my bug request : https://zh.wikipedia.org/wiki/Wikipedia:VPM#.E6.9C.89.E9.97.9C.E9.98.B2.E6.BF.AB.E7.94.A8.E9.81.8E.E6.BF.BE.E5.99.A8 [13:23:07] ;( [13:23:28] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [13:28:01] New review: Hashar; "This right is not available yet on the wmf cluster. We need to wait for the next deployment :-D" [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/11848 [13:29:40] New patchset: Hashar; "(bug 37679) zhwiki let rollbacker abusefilter-log-private" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11848 [13:29:47] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11848 [13:30:19] New patchset: Hashar; "(bug 37679) zhwiki let rollbacker abusefilter-log-private" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11848 [13:30:25] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11848 [13:30:53] New review: Hashar; "I am deploying the configuration nonetheless. Will have to wait for the new AbuseFilter extension to..." [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11848 [13:31:02] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11848 [13:32:40] hashar: were these imagescaler changes tested? [13:33:09] obvious for the merge from production -> test I'm keeping what's in test, but it would be good to know the changes work :) [13:33:50] which one ? [13:34:00] I did send some changes in test for Precise [13:34:06] that is currently running on labs :-D [13:34:42] does it work for the lucid instances too? [13:34:55] they should [13:34:57] untested though [13:34:58] heh [13:35:00] ok [13:35:07] but reviewed by paravoid so we are safe somehow [13:35:09] if I break everything I'll just blame it on you [13:35:16] yeah sure :-] [13:35:29] make sure to write a blog post about collaboratively breaking the site :-]]] [13:35:34] will be glad to share the frontage with you [13:35:36] *grin* [13:35:41] seriously, : [13:35:42] * b17da88 - (HEAD, gerrit/test, test) cronspam sprint - srv222 - use -ignore_readdir_race with find to avoid errors for missing files ( [13:35:44] * 885267c - RT #3117 phase out wikimedia-fonts package (4 days ago) [13:35:45] * 3a6d7bb - Explicitly define fonts package for Precise (4 days ago) [13:35:46] * 9722c0d - 'gs' package renamed 'ghostscript' in Precise (5 days ago) [13:35:47] * 2d04bd2 - enable imagescaler classes on Precise hosts. (5 days ago) [13:35:53] all of them should work on Lucid fine [13:35:55] :) [13:35:59] I have checked the packages manually on a lucid box [13:36:05] * Ryan_Lane nods [13:36:14] I had to do those changes for Precise cause ubuntu renamed several packages [13:36:14] this mwfatallog stuff.... [13:36:22] * 610f4b1 - Ensure that /a exists on imagescalers (4 weeks ago) [13:36:23] [13:36:30] that one I am not sure we want it in production though [13:36:37] that probably need to be something else [13:37:55] I wonder if I'm about to push in a million changes since I didn't do --no-ff [13:38:23] New review: Hashar; "Hmm that patch was adding NS_MAIN, NS_FILE, NS_TEMPLATE" [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/11839 [13:39:00] New patchset: Hashar; "(bug 37675) ruwiki: enable review on Portal:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11839 [13:39:06] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11839 [13:40:09] New review: Hashar; "Patchset 2:" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/11839 [13:40:33] what's this gsbmonitoring stuff? [13:41:28] RECOVERY - Host search23 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms [13:42:50] cmjohnson1: okie dokie [13:44:02] New patchset: Hashar; "(Bug 35712) Add an alias to Help namespace in ml.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11840 [13:44:08] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11840 [13:44:16] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11840 [13:44:18] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11840 [13:45:04] hasher: is there an issue with the indenting of my change? [13:45:26] looks weird on Gerrit, but it is correct on my text editor [13:46:47] New review: Hashar; "If the community is unsure if they want this change, you might want to abandon it and reopen it when..." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/11150 [13:47:10] *ping wrongly, hashar [13:47:11] notpeter: any idea what this gsbmonitoring stuff in the test branch is? [13:48:40] Hydriz: pong [13:48:40] I'm going to assume that the gsbmonitoring stuff was removed from head [13:48:48] see ^ [13:48:51] from production, that is [13:49:01] 1-2 lines above [13:49:13] Hydriz: which change ? [13:49:22] 11840 [13:49:31] Hydriz: the initialisesettings.php files have a weird indenting anyway [13:49:34] it is a mix of tabs and spaces :-( [13:49:35] because it was targetting spence [13:49:46] heh [13:49:55] but we should be using tabs, isn't it? [13:50:08] the vim configuration says to use spaces [13:50:16] lol [13:50:20] I end up using whatever indent is used around the lines I edit [13:50:51] I'm going to be really annoyed if I do this merge and have to redo it because I forgot —no-ff [13:51:00] anyway its merged :P [13:51:59] New patchset: Hashar; "(Bug 36895) Restrict media upload to autopatrolled users on sv.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11837 [13:52:05] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11837 [13:52:17] New review: Hashar; "Patchset 3 is just a rebase" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11837 [13:52:20] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11837 [13:52:49] - Introduce AFTv5 lottery [13:52:50] yeahhh [13:53:17] lets merge all those changes [13:54:07] \o/ [13:56:58] New patchset: Hashar; "(bug 36972) activate the patroller group on nn.wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11150 [13:57:04] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11150 [13:57:22] New review: Hashar; "Patchset 2 is just a rebase." [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11150 [13:57:28] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11150 [13:58:09] lets blank page! [14:00:14] mark, paravoid: http://pastebin.com/cr8qnDFg [14:00:23] I know the geoip stuff was added and should be kepy [14:00:25] kept [14:00:38] what about the purge decommissioned hosts exec? [14:00:53] looking [14:01:13] never seen this before :) [14:01:18] not sure if it was added to test or removed from production [14:02:21] that should still be there [14:02:46] it isn't [14:02:49] in production [14:02:54] Ryan_Lane: no idea [14:03:03] why the hell not [14:03:11] perhaps elsewhere in a different class? [14:03:18] lemme see [14:03:24] not [14:03:30] it's not there at all in prod [14:03:33] grr [14:03:33] only in test [14:03:56] trying to see where it was removed [14:06:49] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [14:08:15] ahhhh [14:08:20] it was turned into a script [14:08:28] git show 1db825bdd703fa94227521c1ef7ab1cd7b0e9661 [14:12:12] I never understand how site.pp ever conflicts [14:15:18]