[00:02:51] New patchset: Dzahn; "removing include base::decommissioned from streber - it's not decom'ed, it actually runs RT, Observium and Rancid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52730 [00:03:03] New patchset: Matanya; "Correct cmd file. removed compatibilty with nagios." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52731 [00:05:03] !log upgrading cp1001 [00:05:09] Logged the message, Mistress of the network gear. [00:06:42] !log installing openjdk/icedtea jre and other package upgrades on gallium (jenkins) [00:06:48] Logged the message, Master [00:07:45] New review: Dzahn; "streber should still live :p" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/52730 [00:07:55] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52730 [00:08:40] kaldari, how's the scap going? [00:08:52] slowly [00:08:54] but surely [00:09:11] Okay, thanks [00:12:41] mutante, is sodium due for any updates? [00:13:21] New patchset: Matanya; "Fixed the path to icinga cmd file, and on nsca." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52726 [00:14:01] Thehelpfulone: nope, just did today [00:14:23] the security ones that is [00:14:42] notpeter: thanks for the lucene search merge, did it broke anything in production ? :-D [00:14:53] yep [00:14:54] ok, so mailman update is still stalled then too I guess until they release something [00:15:05] there was a small problem with the hash strucutre [00:15:12] so puppet was broken on the eqiad nodes [00:15:15] but it was an easy fix [00:15:40] great! [00:15:44] (almost great) [00:15:47] eh [00:16:13] I still have to hack up some iptables rule in beta [00:16:13] in a patchset that big, some peice of syntax is likely to get dropped [00:16:14] was no big deal :) [00:16:35] the instances can't access a public address hosted on labs. The OAI inc updater is thus blocked :-D [00:19:03] Thehelpfulone: are you after 2.1.14-3 specifically, or just in general... [00:19:44] !log kaldari Finished syncing Wikimedia installation... : [00:19:49] Logged the message, Master [00:20:16] kaldari, done? :) [00:20:48] in general mutante, the current version is more than 3 years old IIRC [00:21:11] New patchset: Matanya; "Fixed the path to icinga cmd file all around." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52726 [00:21:42] I wonder if they fixed the issues with chrome yet too [00:22:35] in 12.04 it's 1:2.1.14-3: [00:23:05] Change abandoned: Matanya; "replaced by I9c5805405aa7a0624f1bf4a5939ae9c902e1baf8." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52731 [00:23:19] We're on 2.1.13 now [00:23:22] yeah that's what https://rt.wikimedia.org/Ticket/Display.html?id=2905 says [00:23:29] lolol [00:23:51] wow, raring has 2.1.15 [00:24:03] 2.1.13 released 2009-12-22 [00:24:19] 2.1.14 2010-09-20 and 2.1.15 2012-06-15 [00:40:36] Yay [00:40:36] git is so diff from svn [00:49:35] New patchset: Matanya; "Fixed the path to icinga cmd file all around +file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52726 [00:55:04] !log killall lighttpd and restart lighttpd and apache on streber [00:55:09] Logged the message, Master [00:55:12] !log deploying change 52592 to wikitech [00:55:17] Logged the message, Master [00:55:27] LeslieCarr: Oooh hah [00:55:37] Looks like we didn't change the hostnames for celsus and constable in the salt regex [00:55:44] Change inbound [00:55:51] ah [00:55:52] :) [00:56:51] !log added Set(@ReferrerWhitelist, qw(rt.wikimedia.org:443 rt.wikimedia.org:80)); [00:56:57] Logged the message, Master [00:57:08] Hmm, no, that's not broken in puppet [00:57:13] Must be broken on the salt master then [00:57:57] Ryan_Lane: In salt, how do I remove a decommissioned (technically, renamed) minion? [00:58:11] It looks like salt is trying to hit both celsus.wikimedia.org and celsus.pmtpa.wmnet now [00:59:52] RoanKattouw: just remove it from the regex [00:59:53] I can delete the key [01:00:05] The regex captures both [01:00:11] that's not an issue [01:00:15] (celsus|constable|lots|of|other|crap)\..* [01:00:17] the old one is gone, right? [01:00:21] So there's no regex change to make [01:00:30] Yeah, celsus.wm.o doesn't exist any more. But celsus.pmtpa.wmnet does [01:00:31] I just need to delete the key, then [01:00:33] OK [01:00:34] ok [01:01:05] I don't see a celsus [01:01:15] oh [01:01:18] mispelled it [01:02:08] ok. old key is deleted and new one is added [01:02:19] Yay [01:02:21] Thanks [01:04:25] yes too close to celsius [01:05:22] New patchset: Reedy; "Remove search::apple-dictionary-bridge from ekrem" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52742 [01:10:35] New patchset: Catrope; "Change testwiki config to be closer to enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52743 [01:11:05] Change merged: Catrope; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52743 [01:12:17] !log catrope synchronized wmf-config/InitialiseSettings.php '20388a06 - Normalize VE config on testwiki' [01:12:23] Logged the message, Master [01:16:24] drdee: ping [01:22:35] New patchset: Dzahn; "remove preilly account from zhen/silver (just cleanup, key was already gone)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52744 [01:23:47] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52744 [01:24:20] mutante: boo [01:24:24] mutante: :-) [01:24:45] heh, that's why i added the "just cleanup" ;) [01:27:29] !log installing package upgrades on ekrem [01:27:34] Logged the message, Master [01:31:17] hi preilly [01:33:27] TimStarling: howdy [01:36:17] TimStarling: How was your trip to the States? [01:36:39] it was fine [01:36:50] pity you weren't in town [01:36:59] there was drinking every night [01:37:49] TimStarling: do you mind if I send a PM [02:28:34] !log LocalisationUpdate completed (1.21wmf11) at Fri Mar 8 02:28:34 UTC 2013 [02:28:44] Logged the message, Master [02:31:53] binasher: https://gdash.wikimedia.org/dashboards/jobq/ :( [02:32:12] doh [02:32:13] probably bad gdash conf [02:33:03] https://gdash.wikimedia.org/dashboards/jobq/ [02:33:32] binasher: maybe it's repeating field:pop [02:33:43] * Aaron|home fixes [02:33:48] yup [02:33:49] *** Exception RuntimeError in PhusionPassenger::Rack::ApplicationSpawner (./lib/graphite_graph.rb:238:in `field': A field called pop already exist for this graph) (process 3456): [02:34:21] sorry i missed that in review [02:37:37] New patchset: Aaron Schulz; "Fixed silly bugs in 265a544737f9aea048eaa3e5b7fd2f55991d089a" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52752 [02:37:55] there we go [02:45:14] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52752 [02:51:45] bah [02:51:51] Aaron|home: two other problems [02:52:02] field names can't have -'s [02:52:48] and cactiStyle() can't be called on completely empty values [02:52:55] !log LocalisationUpdate completed (1.21wmf10) at Fri Mar 8 02:52:54 UTC 2013 [02:53:00] Logged the message, Master [02:53:43] so job-insert-duplicate shouldn't be in there until stats are actually being logged [03:01:15] Aaron|home: fixing [03:01:41] yeah I already did the - thing [03:01:52] I was just wondering why there would be 0 job-insert-duplicate items [03:02:01] then looking through code for bugs [03:02:12] binasher: what if a dummy job-insert-duplicate item is sent? [03:02:20] (e.g. 1) [03:03:39] New patchset: Aaron Schulz; "Removed disallowed hyphens from graphite field names." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52757 [03:03:44] New patchset: Asher; "remove non alpha chars from field names, commenting out stats.job-insert-duplicate.count for now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52758 [03:03:44] doh [03:04:25] Change abandoned: Aaron Schulz; "raced out :)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52757 [03:04:32] Aaron|home: afaik, job-insert-duplicate is still getting sent as 0 often [03:04:59] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52758 [03:05:04] wfIncrStats() won't send if it's 0 [03:05:26] oh good! [03:07:13] Aaron|home: graphite does have a treat null as 0 function, which might work here [03:07:25] that would be preferable to sending dummy values [03:07:35] but [03:08:01] might as well wait til the function is working and sending useful stats before including it [03:12:40] binasher: though you don't know if it's working without logging ;) ... though I guess it could use some .log file [03:12:58] though a UDP log for this felt a bit heavier [03:14:29] Aaron|home: you can still check on it via graphite without adding to the main job dash tho [03:14:31] http://bit.ly/12ys5T1 [03:15:08] when that doesn't throw an empty sequence error, you've got data :) [03:16:48] oh right, of course, I was looking at that a few days ago and forgot [03:24:41] binasher: I like how Setup.php-memcached is 22% of thumb.php [03:27:29] Aaron|home: wee! so.. when i changed wgObjectCaches['memcached-pecl'] to just a persistent connection to localhost, it had a big impact in Setup.php-memcached time actually [03:52:17] Aaron|home: the author of twemproxy wouldn't mind dropping by the office tomorrow afternoon, are you going to be around? [03:52:54] if so, i'll see if he wants to come by for a drink around 4 [04:01:41] !log reedy synchronized php-1.21wmf11/extensions/Translate [04:01:43] Get him drunk, steal the source code? ;) [04:01:47] Logged the message, Master [04:05:50] hahahah [06:21:40] !log updated luasandbox to version 1.4 (with pauseUsageTimer) [06:21:46] Logged the message, Master [07:45:58] New review: Nikerabbit; "Shouldn't this be abandoned?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44715 [08:51:32] may I have a wikimedia.org e-mail account? [08:51:36] I'm from the Analytics team [08:55:20] you have to contact office IT for that [08:55:35] I think via your manager [09:16:05] paravoid: ok [09:41:45] New patchset: Matanya; "New ubuntu icon" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52778 [09:48:08] hi can someone tell me how to push into operations/puppet [09:48:15] jeremyb_ ^ :P [09:48:28] remote: You are not allowed to perform this operation. [09:48:29] remote: To push into this reference you need 'Push' rights. [09:48:30] remote: User: petrb [09:48:31] remote: Please read the documentation and contact an administrator [09:48:32] remote: if you feel the configuration is incorrect [09:48:49] I feel that I need to contact an administrator [09:51:16] New patchset: Petrb; "nrpe changed the IP of master server for labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52779 [09:51:36] petan: I'm an admin meta, would that help you? :P [09:51:51] lol no, I found out [09:51:53] git push-for-review-production [09:51:55] that's it :P [09:52:31] paravoid can you merge it? pls [09:54:30] @seen Ryan_Lane [09:54:30] petan: Last time I saw Ryan_Lane they were quitting the network with reason: Quit: Leaving. N/A at 3/8/2013 8:46:50 AM (01:07:40.1836700 ago) [09:54:42] 1 hour :/ [10:04:04] New patchset: awjrichards; "Update X-CS handling to new k/v pair spec" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52606 [12:18:20] New review: Mark Bergsma; "You should use the subnet prefix lists from network.pp instead." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50441 [12:26:46] New patchset: Mark Bergsma; "Revert "Restart bits varnish on cache allocation failures"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52792 [12:28:10] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52792 [12:28:35] New patchset: Mark Bergsma; "Revert "Revert "Restart bits varnish on cache allocation failures""" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52793 [12:30:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52793 [12:31:28] New patchset: Mark Bergsma; "Disable the restart cron job, for now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52794 [12:32:17] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52794 [13:19:40] New patchset: Aude; "(bug 45892) add second cron job for wikidata dispatching changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52797 [13:36:46] New review: Daniel Kinzler; "Can be done, but I'm not sure whether having two processes running continuously is going to speed th..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52797 [13:54:03] Thehelpfulone you have access to RT wtf? [13:56:09] petan: welcome to the club. [13:56:35] ? [13:56:37] which club [13:56:58] you can access certain issues via email, afaik. [13:57:01] *RT [13:57:05] the-club-of-people-who-are-surprised-that-THO-has-rt-access [13:57:09] aha [13:57:17] why he has it? he works for foundation? [13:57:35] o.o [14:00:14] i guess he's just super helpful. [14:20:10] or he's the one who signed more things like non-disclosure agreement [14:36:54] Change abandoned: Ottomata; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49678 [14:46:26] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [15:07:27] New patchset: Ottomata; "Fixing path to api.wsgi" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52803 [15:07:54] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52803 [15:22:41] hi mark, [15:22:46] i'm adding a domain for E3 [15:22:51] wikitech says to get a review, so! [15:22:54] +metrics 1H IN CNAME stat1001 [15:22:57] s'ok? [15:23:06] see also /tmp/dns.diff on sockpuppet [15:27:38] RobH, you up and at 'em? can you approve that? [15:27:40] ok [15:27:47] ah, sorry, marks there :) [15:29:00] ok == review ok? [15:29:03] mark? [15:33:04] ok, its so simple I think ok == ok. [15:34:04] yes ;) [15:34:51] !log adding metrics.wikimedia.org domain, running authdns-update on dobson [15:34:57] Logged the message, Master [16:03:17] New patchset: Ottomata; "Renaming metrics-api to metrics." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52805 [16:05:49] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52805 [16:35:18] New patchset: Ottomata; "Fixing comma in settings.py" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52809 [16:36:35] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52809 [16:52:07] New review: Jeremyb; "$ ldapsearch -v -x -y foo -D bar -b 'ou=hosts,dc=wikimedia,dc=org' \" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/52779 [16:54:47] paravoid can you review my chagne [16:55:52] ahhhh, the eternal paravoid question [16:56:00] haha [16:57:18] or if someone else can do that... [16:57:33] petan, what change? I can look [16:57:46] https://gerrit.wikimedia.org/r/#/c/52779/ [16:58:42] ok, i can merge that, do you know where I need to run puppet and how to check if it works? [16:59:00] you don't need to run puppet anywhere, it's labs so it runs every like 10 minutes [16:59:03] ok cool [16:59:22] it will either work or it will be still broken :) [16:59:25] ok [16:59:25] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52779 [16:59:29] yay! [17:01:07] 10 mins? that sounds like a lot [17:01:58] jeremyb_ I don't know I guess because there is always last puppet ran 10 minutes ago :P [17:02:22] well it must be less than that sometimes? [17:02:27] and i bet more sometimes [17:02:29] probably [17:02:51] the puppet default (OOTB) is 30 mins [17:03:06] idk if ryan made labs special but i doubt it [17:25:31] petan, mailman work and general stuff between RT and Bugzilla etc [17:25:45] and yeah, you need an NDA for an RT account [17:26:51] New patchset: Ottomata; "Adding puppet-merge for sockpuppet puppet merges." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [17:28:25] New patchset: Ottomata; "Adding puppet-merge for sockpuppet puppet merges." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [17:30:51] New review: Ottomata; "This patchset uses the remote branch for merge commands, rather than FETCH_HEAD." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50452 [18:24:40] drdee: So I am going to spin up a new locke replacement for you today [18:25:38] I was just going to have it copy the site.pp entry for locke (new server is europium in eqiad) [18:25:51] awesome [18:26:03] i assume my best judgement on partitioning is fine [18:26:08] as this seems to be normal filesystem use [18:26:47] if its wrong when done, meh, we can reinstall =] [18:27:03] (unless you know a specific setup for paritioning you guys need) [18:27:13] i think this is all default stuff [18:27:17] nothing fancy [18:27:55] cool [18:38:06] drdee: so does this locke replacement need a public IP? [18:38:19] I see that locke has one, but its a udp log aggregator right? [18:38:33] ie: if it doesnt need public IP, safer to not use one. [18:39:16] * YuviPanda sees /topic [18:39:21] * YuviPanda pokes Jeff_Green [18:39:28] hey Jeff_Green! Can I poke you with https://rt.wikimedia.org/Ticket/Display.html?id=4687 [18:39:30] ? [18:39:37] * Jeff_Green ducks & covers! [18:39:39] looking [18:39:43] YuviPanda: it wont happen for 3 days ;] [18:40:10] oh, right. the 3 day waiting period for comments. [18:40:11] access request policy is to leave all requests pending for 3 working days so anyone who wishes to object can do so. [18:40:47] hmm, okay. I was not sure if it just slipped people's radar or something else [18:40:51] in this case I guess it is 'something else' :) [18:41:03] I really wish RT had a dropdown for template replies [18:41:16] so we could use a boilerplate 'approval seems ok but is awaiting the 3 days' [18:41:18] or soemthing [18:41:21] totally [18:41:49] hrmm, i wonder if ticket move to queue can fire a script [18:41:52] * RobH goes to check [18:42:11] RobH: yeah, that would be nice :) [18:44:46] looks like i can, going to mess with it. [18:45:12] RobH: sweet, thanks :) [18:45:21] I suppose I'll poke again on Monday [18:48:17] RobH: about public ip, maybe you can check that with ottomata [18:51:10] * bawolff tries to politely nag ops folks about en.planet being broken for several days now [18:51:32] ahh, its not running updates [18:51:42] bawolff: not sure who in ops actually reads it ;] [18:51:58] I'll try to take a poke at it later and see if I can figure it out [18:52:07] bawolff: thx for the info =] [18:52:31] Thanks :) [18:54:20] woooooo, it works [18:54:28] Jeff_Green: So since you are on RT call, you should know this [18:54:36] i just enabled auto reply when any ticket is moved to access requests [18:54:45] it will fire off the boilerplate email reply i put in that queue [18:54:55] look at rt4700 if you wanna see [18:55:24] RobH: ah cool [18:56:01] Btw the planet thing has a bug at https://bugzilla.wikimedia.org/show_bug.cgi?id=45806 [18:56:57] noted [19:05:12] paravoid: ping [19:05:23] or is there a partman pro in the house? [19:05:50] pro no, but I've worked with it some. what's up? [19:09:08] Jeff_Green: know how to craft a recipe that partitions one device one way, with one set of mount points, and another device another way, with other mount points, with no raid or anything tying the two devices together? [19:09:55] we have no examples of this amongst our partman recipes, and notpeter isn't sure if possible [19:09:59] maybe. i think i may have done that in frack. checking [19:10:56] thanks! [19:11:45] bawolff: So I was looking at it, but then the guy who works on it more than me walked in [19:11:48] so its being looked at =] [19:13:34] cmjohnson1: uhhh [19:13:40] cmjohnson1: wmf4084/europium [19:13:48] yeah...looking at ticket now [19:13:54] if you are near it, can you tell me what port its in? [19:14:00] easier to ask than run the mac check. [19:14:04] i labeled it [19:14:38] show interfaces descriptions | grep wmf4084 [19:14:41] didnt show it [19:14:41] but [19:14:48] show interfaces descriptions | grep 4084 did [19:14:50] bleh. [19:14:56] cmjohnson1: found it ;P [19:14:57] heh [19:15:00] cool [19:16:29] New patchset: Pyoungmeister; "db-pmtpa: further hobbling pmtpa for db dumps" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52885 [19:16:55] hobble on [19:17:51] New patchset: Cmjohnson; "Adding new mac to db1032 to reflect h/w change" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52886 [19:18:09] bawolff: updating en planet .. [19:18:15] mutante: https://bugzilla.wikimedia.org/show_bug.cgi?id=45806 [19:18:31] Change merged: Pyoungmeister; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52885 [19:18:43] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52886 [19:19:32] binasher: I could swear I did this at some point--it may have been on the virtualbox cluster I was working on on my laptop. [19:20:01] I know you can specify a disk to partition like "d-i partman-auto/disk string /dev/sda" [19:20:01] RobH: bawolff: hmm.. this is the issue "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 32: ordinal not in range(128) [19:20:07] New patchset: Asher; "partman for rdb" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52888 [19:20:08] now that's a new one...looking [19:20:08] !log py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave' [19:20:14] Logged the message, Master [19:20:54] binasher: I'm starting to get this feeling about these mysql dumps... http://xkcd.com/349/ [19:21:20] notpeter: I have seen your change to the rsyslog/syslog-ng madness in base.pp [19:21:33] paravoid: still copying deleted files and timeline files [19:21:44] as soon as the former finishes, I'll start the sync [19:21:59] notpeter: still have to figure out if that will fix the issue though since in labs we can not pass a parameter to puppet class :/ [19:22:03] binasher: http://ubuntuforums.org/showthread.php?t=1783522 [19:25:01] that thread seems like a fail [19:25:19] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52888 [19:25:19] Hey Jeff_Green, since you are RT guy [19:25:27] hashar: :(((( [19:25:34] who do you think shoudl review this? [19:25:35] https://gerrit.wikimedia.org/r/#/c/49710/ [19:25:55] binasher: indeed it does. in my experience partman is a narrow ledge on a mountain of fail [19:26:05] notpeter: though we can use a role for that; but then we will have no way to alter the call in base.pp :/ [19:26:22] ottomata: looking [19:26:31] Mutante that sounds odd. Im sure there has been plenty of non ascii data on planet before [19:27:01] * RobH is registered for hackathon. [19:27:49] hashar: yeah.... ugh [19:27:57] then perhaps your way is better for now [19:28:02] until we can do some mass cleanup [19:32:04] RobH, re locke -> europium [19:32:20] yea, can i i make it internal IP? [19:32:22] i think it doesn't need a public IP, except for the fact that it is a udp2log host [19:32:28] which i think means that it needs to be public [19:32:37] so that amsterdam could potentially send logs [19:32:47] "I'm high" [19:32:48] binasher can maybe comment on that [19:32:56] seems like a bad reason to leave it public ip [19:33:03] dont we send udp logs across from esams on another host? [19:33:23] maybe we dont cuz we only cache there and they inherently have public IPs. [19:33:24] i dunno, i just know that we wanted to set analytics1001 up as able to receive these logs and event logs [19:33:32] and binasher said that we ahd to do that on a machine with a public IP [19:33:55] i thought we moved that to internal IP? [19:34:01] !log added Babel extension to wikitech [19:34:07] Logged the message, Master [19:34:51] notpeter: maybe yeah :( [19:34:56] RobH: host's in esams can't access 10.(0|64)/16 [19:35:05] notpeter: ideally we would want to convert the syslog-ng configuration to rsyslog. That will nicely solve the issue. [19:35:15] though they will be able to in a couple months i think, check with LeslieCarr [19:35:18] New patchset: Demon; "Removing 444 unused imports" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/52890 [19:35:18] New patchset: Demon; "Update gitignore" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/52891 [19:35:18] New patchset: Demon; "Added 84 serialVersionUID" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/52892 [19:35:18] New patchset: Demon; "Close a couple of leaking resources" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/52893 [19:35:19] New patchset: Demon; "Fix a bunch of type mistakes that relied on casting" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/52894 [19:35:19] New patchset: Demon; "Useless warning supression" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/52895 [19:35:19] notpeter: if anyone is willing, it can even be tested on beta [19:35:20] yea but ideally we would have some singe point of UDP logging entry host [19:35:22] but meh. [19:35:35] more spofs? sounds great [19:35:50] ideally they will be ... [19:36:00] bawolff: yea, especially since others like Russian don't have that problem, trying to identify the evil feed [19:36:59] ok, so yea [19:37:02] ottomata: will make it public ip [19:37:12] the saga of 'how we handle udp logging' is not a fight i want to take on today. [19:37:19] (or ever) [19:37:21] haha [19:37:30] i think we're all still recovering from last week [19:37:36] haha, ok, RobH, thanks [19:37:41] i slept 4 hours due to insulin shennanigans [19:37:46] im barely coherent. [19:38:04] ^ today or always? [19:38:07] Thehelpfulone: looks like it worked! [19:38:23] Mutante: I have to go. Thanks for looking into thr issue :-) [19:38:23] cmjohnson1: a little from column a, a little from column b [19:41:56] Thehelpfulone: i dont see anyplace to actually change the from. [19:42:18] ahh, queue description [19:44:21] ottomata: has this limn stuff been tested in labs or similar? [19:49:18] New patchset: Matmarex; "(bug 42413) Set $wgCategoryCollation to 'uca-pl' for the Polish Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/51183 [19:49:18] New patchset: Matmarex; "(bug 45444) Set $wgCategoryCollation to 'uca-uk' for the Ukrainian Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52901 [19:49:18] New patchset: Matmarex; "(bug 45446) Set $wgCategoryCollation to 'uca-sv' for the Swedish Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52902 [19:49:19] New patchset: Matmarex; "(bug 45911) Set $wgCategoryCollation to 'uca-pt' for the Portuguese Wikipedia and Wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52903 [19:52:49]