[00:00:13] the major related fix in that history is the switch to an updated/backported jemalloc implementation, the allocator in -wm16 has definite bugs [00:00:52] so way would cp1068 act so nicely compared to cp1065/1066? [00:01:02] didn't get the right triggers? [00:01:08] so, right now is when the Lightning Deploy window starts, we have 3 people lined up for it, what say you, bblack, on timing? [00:01:09] voodoo [00:01:21] as a rule all complex software always has bugs, it's just a question of whether you've managed to trigger them yet or not [00:01:58] ok, please reboot cp1066 when you feel like :) [00:02:03] greg-g: varnish restarts are fast and this should be an independent issue given lb [00:02:33] should, yeah, and we have plenty of head room, it seems, so you feel comfortable with that going on while you diagnose/upgrade? [00:03:12] person doing current diagnosis on production has say over LD, in my book ;) [00:03:19] much better now : https://ganglia.wikimedia.org/latest/graph_all_periods.php?h=cp1066.eqiad.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1389830552&g=cpu_report&z=large&c=Text%20caches%20eqiad [00:03:24] yes [00:03:40] superm401: ok, with that, you're up [00:03:44] superm401: fire when ready [00:03:57] mutante: because varnish isn't running :) [00:04:10] i guessed that [00:04:30] greg-g, thanks, will do. [00:04:42] TimStarling sounds like he wants to gather more data before reboot, I assume he'll issue the reboot when he's ready [00:05:00] yeah, just quickly checking the syslog [00:05:06] bblack, okay, should I wait? [00:05:07] then I'll do a shutdown -r +1 [00:05:19] it shouldn't affect deployments should it? [00:05:23] I think paravoid has the best handle on the XFS/kernel/alloc related issues currently, he's already got some data and theories on it [00:05:36] superm401: I don't think you need to wait [00:06:02] syslog is full of fail [00:06:06] :) [00:06:11] Jan 15 23:34:15 cp1066 kernel: [15148842.004005] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [00:06:11] PROBLEM - Puppet freshness on rhodium is CRITICAL: Last successful Puppet run was Wed 15 Jan 2014 06:04:59 PM UTC [00:06:50] TimStarling: dec 2: 16:33 paravoid: rebooting cp1065, usual XFS deadlock [00:06:55] bblack, TimStarling, alright, will go. [00:06:59] it was logging that once every 2 seconds [00:07:44] !log rebooting cp1066 following XFS deadlock [00:07:50] Logged the message, Master [00:07:50] makes for a good log message at least [00:09:53] (03CR) 10Mattflaschen: [C: 032] Enable GuidedTour on translated languages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105997 (owner: 10Reza) [00:10:02] (03Merged) 10jenkins-bot: Enable GuidedTour on translated languages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105997 (owner: 10Reza) [00:10:21] PROBLEM - Host cp1066 is DOWN: PING CRITICAL - Packet loss = 100% [00:11:41] RECOVERY - Host cp1066 is UP: PING OK - Packet loss = 0%, RTA = 4.99 ms [00:12:41] RECOVERY - Varnish HTTP text-frontend on cp1066 is OK: HTTP OK: HTTP/1.1 200 OK - 197 bytes in 0.000 second response time [00:15:07] the backend one was bitten by the mmap-address thing and didn't start on its own, it's started now [00:15:21] RECOVERY - Varnish HTTP text-backend on cp1066 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.001 second response time [00:15:35] peace is back [00:16:16] !log mflaschen synchronized wmf-config/InitialiseSettings.php 'Deploy GuidedTour to astwiki, fawiki, and ruwiki' [00:16:23] Logged the message, Master [00:17:53] greg-g: Does that ---^^ mean LDs are a go now? [00:18:00] RoanKattouw: yeah [00:18:06] Cool [00:18:11] RoanKattouw: was goign to let you know when superm401 was done [00:18:18] Let me start prepping now because Gerrit is super slow today [00:18:29] RoanKattouw, I have a couple more things. [00:18:30] * ^d already has commands waiting for merges & pulls [00:18:32] The hotfix [00:18:37] :) [00:18:38] * ^d is waiting for his turn [00:18:45] so many people itching at their turn ;) [00:20:06] <^d> Better people waiting than us all saying that we started scap at the same time. [00:20:28] ^d++ [00:20:55] (03PS1) 10Mwalker: Qualified Lookup of OCG Class [operations/puppet] - 10https://gerrit.wikimedia.org/r/107742 [00:21:33] (03CR) 10Ori.livneh: [C: 032 V: 032] Qualified Lookup of OCG Class [operations/puppet] - 10https://gerrit.wikimedia.org/r/107742 (owner: 10Mwalker) [00:26:56] (03PS1) 10Mwalker: Introducing the role:ocg:production [operations/puppet] - 10https://gerrit.wikimedia.org/r/107744 [00:27:29] (03CR) 10Ori.livneh: [C: 032 V: 032] Introducing the role:ocg:production [operations/puppet] - 10https://gerrit.wikimedia.org/r/107744 (owner: 10Mwalker) [00:29:04] !log mflaschen synchronized php-1.23wmf9/extensions/GettingStarted/ 'Sync GettingStarted wmf9 for hotfix' [00:29:10] Logged the message, Master [00:29:23] done? [00:29:27] !log mflaschen synchronized php-1.23wmf10/extensions/GettingStarted/ 'Sync GettingStarted wmf10 for hotfix' [00:29:33] Logged the message, Master [00:29:37] RoanKattouw, ^d, greg-g, done [00:30:03] superm401: confirmed all's good? [00:30:17] greg-g, no, I'll test now. [00:30:31] (03PS6) 10Physikerwelt: added basic hbase support [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 [00:31:06] (03PS1) 10Mwalker: Apparently Cscott does not have shell [operations/puppet] - 10https://gerrit.wikimedia.org/r/107745 [00:31:51] * RoanKattouw runs git review and goes to grab a drink [00:31:54] (03CR) 10Ori.livneh: [C: 032 V: 032] Apparently Cscott does not have shell [operations/puppet] - 10https://gerrit.wikimedia.org/r/107745 (owner: 10Mwalker) [00:33:06] greg-g, deployment looks good. [00:33:48] cool [00:37:12] <^d> RoanKattouw: I made repos faster. [00:37:13] <^d> gc <3 [00:38:12] (03PS1) 10Mwalker: Add standard role to rhodium [operations/puppet] - 10https://gerrit.wikimedia.org/r/107748 [00:38:38] (03CR) 10Ori.livneh: [C: 032 V: 032] Add standard role to rhodium [operations/puppet] - 10https://gerrit.wikimedia.org/r/107748 (owner: 10Mwalker) [00:39:06] Maybe less slow? [00:39:13] git review on those two wmf/* commits still took a while [00:39:18] And the VE repos are still slow as molasses [00:39:38] Alright, my turn to go [00:40:11] RECOVERY - Puppet freshness on rhodium is OK: puppet ran at Thu Jan 16 00:40:01 UTC 2014 [00:40:21] (Seriously though I have the feeling there's more going on here than just gc, fetches are suddenly much slower today than they were on Monday) [00:41:03] Maybe it's something on the office network [00:41:07] Cause it's fast from tin [00:42:15] RoanKattouw: are you using ssh or https? are you possibly tunneling through another host without realizing it? [00:42:24] ssh [00:42:28] And I don't think so [00:43:21] !log catrope synchronized php-1.23wmf9/extensions/VisualEditor 'Update VE for cherry-picks' [00:43:26] Logged the message, Master [00:43:32] ssh -v confirms it's not tunneling [00:43:38] !log catrope synchronized php-1.23wmf10/extensions/VisualEditor 'Update VE for cherry-picks' [00:43:44] Logged the message, Master [00:43:56] RoanKattouw: maybe tcpdump to see where it hangs? [00:44:02] * greg-g has to run, enjoy the deploys [00:44:10] debug1: Connecting to gerrit.wikimedia.org [2620:0:861:3:208:80:154:81] port 29418 is where ssh -v hangs [00:44:13] * RoanKattouw fires up wireshark [00:44:49] Oh! Maybe it's because it's IPv6 [00:45:16] Yup [00:45:17] It's IPv6 [00:45:39] ^d: The Gerrit slowness is not your fault, it's the office's IPv6 being broken [00:45:57] huh, I was going to jokingly blame it, but wow [00:45:59] <^d> +1 for wfh! [00:47:04] <^d> RoanKattouw: You all done? [00:47:36] ^d: Yes, sorry, go for it [00:47:40] <^d> No worries :) [00:47:41] Got distracted writing an email about the IPv6 breakage [00:47:48] (03CR) 10Chad: [C: 032] Add new CirrusSearch logs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107503 (owner: 10Chad) [00:47:55] (03Merged) 10jenkins-bot: Add new CirrusSearch logs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107503 (owner: 10Chad) [00:48:59] !log demon synchronized wmf-config/InitialiseSettings.php 'Config new Cirrus logs' [00:49:05] Logged the message, Master [00:49:06] (03PS1) 10Mwalker: Better (12.04) packages for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/107751 [00:50:56] (03PS2) 10Mwalker: Better (12.04) packages for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/107751 [00:51:12] (03CR) 10Ori.livneh: [C: 032 V: 032] Better (12.04) packages for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/107751 (owner: 10Mwalker) [00:53:11] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [00:53:13] ^d: so is gerrit broken? [00:53:20] <^d> No. [00:53:21] it still takes years to push anything [00:53:26] (03PS6) 10BryanDavis: Add logstash config for udp2log [operations/puppet] - 10https://gerrit.wikimedia.org/r/106154 [00:53:32] <^d> See what Roan said about ipv6? [00:54:26] AaronSchulz: gerrit.wm.o times out over IPv6 from the office network [00:54:26] So every time, it tries IPv6, times out after 127 seconds, then falls back to IPv4 [00:54:26] Breakage seems recent because it worked for me just last night [00:54:43] And Gerrit IPv6 isn't broken because I used it from the conference wifi in Australia last week and it was fine [00:55:43] <^d> Gerrit's had ipv6 for several months now. [00:56:54] * RoanKattouw edits /etc/hosts to get gerrit IP to IPv4 [00:58:38] WFM at home on ipv6.. [00:59:06] RoanKattouw: 'AddressFamily inet' in the relevant ~/.ssh/config entry [01:00:34] !log aaron synchronized php-1.23wmf10/includes/filebackend/SwiftFileBackend.php '1218cc7e9cba8342d0184523f1d7c1fec608e656' [01:00:41] Logged the message, Master [01:01:37] ori: https://gerrit.wikimedia.org/r/#/c/107752/ silly fix [01:02:22] AaronSchulz: silly merge [01:09:34] ori: Nice, thanks [01:26:53] !log catrope synchronized php-1.23wmf9/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' [01:27:00] Logged the message, Master [01:27:48] !log catrope synchronized php-1.23wmf10/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWTransclusionNode.js 'touch' [01:27:54] Logged the message, Master [01:28:10] !log catrope synchronized php-1.23wmf10/resources/startup.js 'touch' [01:28:16] Logged the message, Master [01:30:21] !log catrope synchronized php-1.23wmf10/extensions/VisualEditor 'Forgot to run git submodule update earlier' [01:30:28] Logged the message, Master [02:19:10] Hi, I'm looking to help out with operations. Do I need to talk to someone and get permissions or is the best way to just jump into bugzilla etc? [02:21:42] Hey there Noodles` :) [02:21:51] Hi :) [02:21:56] There are probably more people around here a bit earlier in the day [02:22:22] Ok, are most people on EST? [02:22:26] One thing you can easily do without having to get permissions assigned to you is to submit changes to our puppet config in Gerrit [02:22:40] There's documentation at https://www.mediawiki.org/wiki/Gerrit about how you can set up an account there [02:23:47] We always welcome puppet cleanup; other than that I don't really know what's been going on in ops land recently, actual ops people should have a better idea of what tasks are open [02:23:58] Unfortunately most of them aren't in Bugzilla, and RT requires login [02:24:20] Ops people are ... all over really [02:24:21] ok great, thanks. Who are the best people to talk to when they're online? [02:24:36] matanya (not on at the moment, but frequently on) is a volunteer that has successfully seen several puppet patches through [02:24:43] There's a bunch on PST here in SF, and a few on EST, CET and EET [02:24:51] Ahm, that's a great question [02:24:54] if you ping him i'm sure he could share his experiences [02:25:07] Yeah matanya is a good person to talk to because he's already doing what you're trying to do [02:25:33] Jeff_Green is on RT duty this week which means he's triaging stuff all the time, so he'll have a good up-to-date overview of outstanding tasks [02:25:43] Thanks, I'll send him a message [02:25:46] Noodles`: you can also look at and find modules / manifests that do not conform to that guide (there are many) [02:26:05] paravoid and mark are also people that tend to know what's going on and what needs to be done [02:26:17] (Jeff is in EST, Mark is in CET, Faidon (paravoid) is in EET) [02:26:38] And matanya is, what, EETish? [02:26:54] UTC+2 [02:27:01] !log LocalisationUpdate completed (1.23wmf10) at 2014-01-16 02:27:00+00:00 [02:27:07] Right, yeah that's EET [02:27:14] I forget where he is but I think it's Israel? [02:27:17] Logged the message, Master [02:28:39] Thanks guys. May I ask what your roles are? Are you both volunteers? [02:29:06] We're both WMF staff [02:29:20] I am a developer on the VisualEditor project, and we've met before at a few Beowulfs [02:29:40] Roan: ipv6 happy for you?> [02:29:55] Ori has walked off into a meeting room but I can tell you he's a performance engineer [02:30:03] cajoel: Yeah working fine now [02:30:08] Roan: for future troubleshooting a quick trace route is gold. (mtr is a nice one..) [02:30:15] Right, yeah, will do [02:30:20] Is this why Ori was restarting gerrit the last few days? (possible) [02:30:26] It took me a while to even figure out IPv6 was the problem [02:30:33] I doubt it because it was nice and snappy just last night [02:30:39] ok [02:30:51] Cool, I'm CTO and run devops at Drugs.com, just looking to learn a bit more and help out [02:30:58] Chad did GC a bunch of repos today to fight the slowness when we still thought it was server-side [02:31:01] another fallback in urgent situations would be to disable your ipv6 stack on your machine. [02:31:24] Noodles`: Hmm, I think I might have you confused with Noodles without the ` then [02:31:27] want me to try ipv6 from here, I have native or was a problem with a specific location? [02:31:28] Him I have met in person [02:31:35] It was a problem with the office network [02:31:38] well, I'm really sorry if it was due to the ipv6 network layer.. I'll put in some ipv6 specific performance metrics in the office. [02:31:38] oh ok, that could get confusing [02:32:00] to track if/when this happens again.. [02:32:12] Noodles`: You wouldn't happen to be from Northern Ireland, would you? [02:32:16] I'll also probably be briefly breaking it again late tonight to troubleshoot with our isp [02:33:11] no, from NZ [02:33:29] Aha :) [02:35:03] I'm sure there's a few "Noodles" around the place :) [02:35:08] Yeah :) [02:36:40] Thanks again, I'll idle here, and hopefully I'll catch someone later [02:51:30] !log LocalisationUpdate completed (1.23wmf9) at 2014-01-16 02:51:29+00:00 [02:51:36] Logged the message, Master [02:56:52] New format is live. [03:00:41] !log Deleted all non-current pacct on neon; activity seems to have increased tenfold and /var/log is OOS [03:00:47] Logged the message, Master [03:03:11] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Thu 16 Jan 2014 12:02:23 AM UTC [03:28:59] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-01-16 03:28:58+00:00 [03:29:05] Logged the message, Master [03:55:36] (03PS1) 10Reedy: Enable Thanks and Echo on zhwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107800 [05:14:31] PROBLEM - mysqld processes on db1042 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [05:15:33] o: [05:17:15] * Reedy blames domas [05:19:55] (03PS1) 10Springle: depool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107801 [05:20:29] (03CR) 10Springle: [C: 032] depool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107801 (owner: 10Springle) [05:20:35] (03Merged) 10jenkins-bot: depool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107801 (owner: 10Springle) [05:21:38] !log springle synchronized wmf-config/db-eqiad.php 'depool db1042' [05:21:44] Logged the message, Master [05:33:31] RECOVERY - mysqld processes on db1042 is OK: PROCS OK: 1 process with command name mysqld [05:49:26] (03PS1) 10Springle: depool db1033 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107803 [05:56:08] (03PS10) 10Physikerwelt: Add Mathoid module (TeX -> MathML / SVG conversion web service) [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 [05:57:42] (03CR) 10Springle: [C: 032] depool db1033 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107803 (owner: 10Springle) [05:58:52] !log springle synchronized wmf-config/db-eqiad.php 'depool db1033' [05:58:58] Logged the message, Master [06:00:18] (03PS1) 10Springle: repool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107804 [06:00:41] (03CR) 10Springle: [C: 032] repool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107804 (owner: 10Springle) [06:00:47] (03Merged) 10jenkins-bot: repool db1042 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107804 (owner: 10Springle) [06:01:28] !log springle synchronized wmf-config/db-eqiad.php 'repool db1042' [06:01:34] Logged the message, Master [06:04:11] PROBLEM - Puppet freshness on neon is CRITICAL: Last successful Puppet run was Thu 16 Jan 2014 12:02:23 AM UTC [06:14:37] (03PS1) 10Springle: remove db29 for decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/107805 [06:16:10] (03CR) 10Springle: [C: 032] remove db29 for decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/107805 (owner: 10Springle) [06:23:04] morning [06:23:47] hi paravoid [06:52:31] RECOVERY - LibreNMS HTTPS on netmon1001 is OK: HTTP OK: HTTP/1.1 200 OK - 6603 bytes in 0.050 second response time [07:00:15] paravoid: Is there a protocol for list admins to reset or reminder owner password? [07:00:35] If this requires ops, could you reset cvn@ and cvn-private@? [07:00:48] and send to -owner (or just me) [07:01:24] I don't do mailman, do you mind filing an RT so someone else can have a look? [07:01:53] throwing a question to the room -- in trebuchet; why did I create a deploy repo? wouldn't it just be easier to have a deploy branch? [07:02:45] mwalker: yes [07:03:11] There were multiple questions. [07:03:37] I tend to do that [07:03:40] it's a bad habit [07:05:28] Krinkle: afaik, its just been "file a bz" [07:05:34] in the past [07:05:37] filed rt [07:06:12] Yeah, Bugzilla would do it too. I think it's just a shared password. [07:06:14] does thehelpfulone or whoever does the MLs these days have access to whichever RT queue you put it? [07:08:01] (03PS1) 10Faidon Liambotis: tcpircbot: fix multiple unreferenced var errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/107807 [07:08:04] ori: wanna have a look? :) [07:09:02] oh fuck [07:09:18] yeah i merged that with the intent of carefully rolling it out and then got distracted [07:09:27] i'll have a look [07:10:18] (03CR) 10Ori.livneh: [C: 032] tcpircbot: fix multiple unreferenced var errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/107807 (owner: 10Faidon Liambotis) [07:10:28] paravoid: much appreciated [07:10:42] oh yeah, i remember why i got distracted [07:10:50] a puppet run was already in progress on neon [07:10:57] because a puppet run is always in progress in neon [07:11:03] *on [07:11:07] yes, it's because naggen is broken with 2.7 :( [07:11:27] naggen is the hack I've done for not making multiple thousand exported resources [07:11:34] it dumps directly from the database into configs [07:11:40] it used to do one sql query [07:11:54] now it does thousands, due to puppet internals having changed [07:12:09] (which is the price we're paying for using something that ties into puppet internals) [07:12:35] I was hoping for puppetdb, but I don't see this happening soon [07:12:40] so maybe I should just rewrite it [07:12:42] maybe in python [07:18:21] RECOVERY - Puppet freshness on neon is OK: puppet ran at Thu Jan 16 07:18:10 UTC 2014 [07:25:59] Krinkle: replied in RT [07:27:43] there's another logmsgbot bug i introduced [07:27:46] fixing [07:28:40] it's because i snubbed Gloria [07:28:48] now I'm paying the price [07:32:33] (03PS1) 10Ori.livneh: tcpircbot: Correctly JSON-serialize array values [operations/puppet] - 10https://gerrit.wikimedia.org/r/107809 [07:33:04] (03CR) 10Ori.livneh: [C: 032 V: 032] tcpircbot: Correctly JSON-serialize array values [operations/puppet] - 10https://gerrit.wikimedia.org/r/107809 (owner: 10Ori.livneh) [07:33:47] (03CR) 10Faidon Liambotis: [C: 04-2] "Packaging single nagios/icinga check sounds like a huge overkill to me..." [operations/debs/check_ganglia] (debian) - 10https://gerrit.wikimedia.org/r/107723 (owner: 10Ottomata) [07:38:38] (03PS1) 10Mwalker: OCG Trebuchet Config and Removal of Node [operations/puppet] - 10https://gerrit.wikimedia.org/r/107810 [07:39:24] ^ If anyone wants to merge and push that -- it would be nifty :D [07:41:23] (03PS2) 10Faidon Liambotis: OCG Trebuchet Config and Removal of Node [operations/puppet] - 10https://gerrit.wikimedia.org/r/107810 (owner: 10Mwalker) [07:41:33] (03PS3) 10Faidon Liambotis: OCG: update trebuchet config and remove Node [operations/puppet] - 10https://gerrit.wikimedia.org/r/107810 (owner: 10Mwalker) [07:41:40] bad ori [07:41:50] (03CR) 10Faidon Liambotis: [C: 032] OCG: update trebuchet config and remove Node [operations/puppet] - 10https://gerrit.wikimedia.org/r/107810 (owner: 10Mwalker) [07:42:00] ugh [07:42:02] fixing [07:43:23] mwalker: merged [07:43:31] thanks! [07:44:33] also bad ottomata for spamming icinga so much for so long [07:44:39] oh he's not here [07:44:52] also paravoid, I'm in the process of testing my pristine backport of texlive 2012 -- if it works (and it doesn't break the math extension); what's the process for importing my PPAs into our local stuff? [07:45:03] 2012? [07:45:05] not 2013? [07:45:42] 2012 is what shipped with stuff up until 13.10 and it looked like it had some interesting dependencies that I didn't want to touch [07:45:53] I can try with 2013; but we've been running under 2012 [07:51:09] arsenic bast1001 fenari gallium hume kaulen lanthanum neon rhodium searchidx1001 searchidx2.pmtpa.wmnet vanadium [07:51:20] are the boxes that have packages named "texlive", excluding all mediawiki appservers [07:51:41] anyway [07:51:53] I really want someone from ops to help you with all that [07:51:56] search... interesting [07:52:38] I thought this was Jeff, but Jeff is pushing back due to time constraints, so maybe someone else? [07:52:52] I'm happy to work with anyone that has time [07:53:09] (another option that was suggested on the mailing list would be to use a local FS repo of the debs that I created) [07:53:14] no :) [07:53:19] that's a terrible idea [07:54:20] (03PS1) 10Ori.livneh: tcpircbot: avoid shadowing parent class's 'channels' attribute [operations/puppet] - 10https://gerrit.wikimedia.org/r/107811 [07:54:49] (03PS2) 10Ori.livneh: tcpircbot: avoid shadowing parent class's 'channels' attribute [operations/puppet] - 10https://gerrit.wikimedia.org/r/107811 [07:54:55] (03CR) 10Ori.livneh: [C: 032 V: 032] tcpircbot: avoid shadowing parent class's 'channels' attribute [operations/puppet] - 10https://gerrit.wikimedia.org/r/107811 (owner: 10Ori.livneh) [07:55:25] paravoid: -b plz [07:55:59] thanks [08:02:14] i need to restart it one last time [08:03:13] I have the most terrible headache, and my clothes are missing. [08:04:06] all done. [08:07:32] (03CR) 10Matanya: added basic hbase support (031 comment) [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [08:09:59] (03CR) 10Physikerwelt: added basic hbase support (031 comment) [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [08:10:29] ori: thank you for yet another ops task ori :) [08:10:55] matanya: are you looking for more TODOs? :) [08:11:27] paravoid: i got some yesterday from mark, but i can add more to my growing list [08:13:41] so the answer is yes, paravoid :) [08:15:04] paravoid, matanya: there was another person on earlier today looking to get involved, Noodles`, so keep an eye out [08:20:08] (03PS7) 10Physikerwelt: added basic hbase support [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 [08:43:02] matanya: oh? what was mark's TODOs? [08:44:20] paravoid: make url-downloader a module, and convert all webserver::* to use the new webserver::apache and on stage 2 convert it into a module, and get rid of the other methods for calling webserver [08:44:36] ok [08:44:52] so the first one is done: https://gerrit.wikimedia.org/r/#/c/107590/ [08:45:03] mine was to write puppet 3 compatibility fixes [08:45:05] mostly scoping issues [08:45:19] now trying to understand the second atm [08:45:58] i'm glad my company uses 3 for some time now, with no issues [08:48:23] we're not very far away [08:48:36] someone just needs to spend 2-3 days fixing a few things [08:48:58] paravoid: wanna do a sprint on that ? [08:51:00] sure, but I was hoping matanya would help first :P [08:52:01] ok. What was the plan ? [08:52:13] fix all the warnings that 2.x is throwing [08:52:13] shoot paravoid [08:52:38] paravoid: mind throwing them on etherpad? [08:55:12] https://etherpad.wikimedia.org/p/Puppet3 [08:55:28] a lot of them are admin.pp ones, so it's really one error [08:56:06] and most of them are trivial [08:57:03] i'll ignore the private for obvious reason [08:57:17] yeah :) [08:57:52] we also have to do the same with labs for manifests that are potentially only applied in labs, but let's start with those first [08:58:09] (03PS4) 10Matanya: etherpad: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107567 [08:58:53] i hope this is what mark meant [09:01:22] matanya: so, what do you think? [09:02:14] paravoid: i'll do one now, and see if that is what you meant? if it is, you will merge it and if not, comment and i'll see what you did mean, works for you? :) [09:02:37] yes! [09:03:57] (03PS1) 10Matanya: webserver.pp : puppet 3 compatibility fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/107814 [09:04:06] here you go paravoid ^ [09:04:59] s/webserver.pp : /webserver:/ [09:05:04] no space before colon, no .pp [09:05:50] also while puppet 3 compatibility fix is right, I'd personally prefer something like "fully qualify variable" or something like that [09:05:57] (03PS2) 10Matanya: webserver: puppet 3 compatibility fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/107814 [09:06:10] next time [09:07:37] ok, so if the patch itself is ok with you, you may merge :) [09:08:26] (03CR) 10Faidon Liambotis: [C: 032] webserver: puppet 3 compatibility fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/107814 (owner: 10Matanya) [09:08:49] so, that kind of thing [09:08:53] some are a bit more complicated [09:08:56] paravoid: i'll start from bottom, you from top [09:10:09] do admins.pp next [09:10:24] i can barely look at the file [09:10:28] heh [09:10:30] my eyes heart [09:10:50] we must add a defined type there one day [09:11:00] yeah... [09:11:22] oh, hrm, admins.pp will be less trivial [09:15:03] why? [09:16:03] morning nuria [09:16:13] holaaa [09:18:58] (03CR) 10Ori.livneh: [C: 032] Call updateBitsBranchPointers in multiversion/switchAllMediaWikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107579 (owner: 10Reedy) [09:19:22] !log ori updated /a/common to {{Gerrit|Id13e614e5}}: repool db1042 [09:19:28] Logged the message, Master [09:30:08] paravoid: would should be done with variables not in classes? [09:30:16] what? [09:30:35] e.g $ganglia_url in manifests/nagios.pp [09:31:25] that can't be used as "${classname::ganglia_url}" [09:32:14] good question :) [09:32:26] that would be $::ganglia_url [09:32:32] but i dislike that [09:32:42] well yeah, but this top-scoped variables are ugly [09:32:48] but this could do it for puppet 3, indeed [09:32:52] then we can do hiera maybe [09:33:30] well if the goal is to stop the warnings and see how much compatible we are with puppet3 [09:33:42] that 'd be ok for now [09:33:52] that is what i did at first, but i found it ugly and risky [09:34:05] ugly yes. risky ? [09:34:37] if someone ever uses this locally it might have scoping issues, unlikely, but might happen [09:35:33] anyway, i'll do it now, but this just encourages me to move more stuff to normal modules. [09:35:41] :-) [09:35:57] yes [09:45:12] (03PS1) 10Matanya: nagios: puppet 3 compatibility fix: fully qualify variables [operations/puppet] - 10https://gerrit.wikimedia.org/r/107819 [09:46:26] (03PS1) 10Mwalker: Would help if the OCG config was in the right spot [operations/puppet] - 10https://gerrit.wikimedia.org/r/107820 [09:46:48] MaxSem: good morning [09:47:00] paravoid, ori: could I get one of you two to review ^ and puppet-merge [09:47:07] hey [09:47:19] morning max [09:51:00] (03CR) 10Alexandros Kosiaris: [C: 032] "LGTM but this will need a rebase (and a manual one at that i am afraid)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98307 (owner: 10Faidon Liambotis) [09:56:33] (03PS1) 10Matanya: hive: puppet 3 compatibility fix: fully qualify variables [operations/puppet] - 10https://gerrit.wikimedia.org/r/107821 [10:04:24] (03PS1) 10Matanya: ldap: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 [10:08:25] (03Abandoned) 10Hashar: beta: pull VisualEditor individually [operations/puppet] - 10https://gerrit.wikimedia.org/r/107575 (owner: 10Hashar) [10:13:25] (03PS2) 10Alexandros Kosiaris: decom hooper,eiximenis [operations/puppet] - 10https://gerrit.wikimedia.org/r/107159 [10:13:35] gah, always think BZ is all done and then more requests to modify https://gerrit.wikimedia.org/r/#/q/status:open+project:wikimedia/bugzilla/modifications,n,z :p [10:14:33] mutante: ahaha. Those are quite a few... [10:14:34] re: backlog sumanah already got her RT pass reset by me, no need to do it again [10:15:42] akosiaris: also without that my gerrit queue has never been so full :) (lintint campaigns etc) [10:15:48] linting [10:16:29] Ah yes, I have started using jon robson's gerrit.py thingy. I find it more useful that the gerrit interface [10:16:30] but should process them by age and get the ancient ones out first:) [10:16:49] akosiaris: ah, didn't know, might take a look [10:17:15] it does exactly what you said... lists them by score and then age [10:17:23] cool [10:17:26] all in all 25 in operations/puppet [10:17:58] better than before the end-of-year cleanup:) [10:18:11] akosiaris: how many of them are mine? :) [10:19:11] matanya: 11 [10:19:28] ok, i'll you a few more :) [10:19:32]