[00:13:27] ebernhardson: $.proxy [00:24:42] hi, I want to request shell access. I was following https://wikitech.wikimedia.org/wiki/Requesting_shell_access but I'm not sure if there's a way to check if I have an RT account or not. I didn't get any message after submitting the password reset form so I'd assume I don't have one.w [00:25:18] You can just email ops-requests [00:25:56] ops-requests@wikimedia.org? OK, I'll update that wiki page [00:34:05] thanks Reedy, sent [00:41:43] (03PS1) 10Gage: Enable varnishkafka on bits caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/109800 [00:41:58] jgage_: \o/ [00:42:50] woo, a lot of reading and a tiny change [00:42:53] sorry that took so long [00:42:58] at least now i understand what i'm doing [00:53:17] <^d> Friendly reminder: gerrit coming down in about ~40mins for update. [00:55:05] (03PS1) 10Springle: repool db1006, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109804 [00:56:04] (03CR) 10Springle: [C: 032] repool db1006, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109804 (owner: 10Springle) [00:56:28] !log ori synchronized php-1.23wmf11/extensions/WikimediaEvents 'Ifc697cbe6: Revert I829790cd5, removing module storage logging' [00:56:35] Logged the message, Master [00:56:56] !log springle synchronized wmf-config/db-eqiad.php 'repool db1006, warm up' [00:57:00] greg-g: FYI we scrubbed enabling Flow on enwiki today. Instead just updating Flow in wmf11 on mediawiki.org [00:57:06] Logged the message, Master [00:57:33] Did Flow go out? [00:57:54] Gloria: no, not on enwiki [00:58:29] spage: I'm looking at https://wikitech.wikimedia.org/wiki/Deployments now. Has it been rescheduled or is it going out now(ish)? [00:58:35] Just curious. [00:59:24] Gloria yeah I'll update it. [01:05:34] !log ori synchronized php-1.23wmf10/extensions/WikimediaEvents 'Ifc697cbe6: Revert I829790cd5, removing module storage logging' [01:05:46] Logged the message, Master [01:10:06] spage: Thanks. :-) [01:11:48] greg-g: https://wikitech.wikimedia.org/wiki/Deployments still has "Week of January 20" on it, BTW. [01:18:42] !log ori synchronized php-1.23wmf11/extensions/WikimediaEvents [01:18:51] Logged the message, Master [01:19:07] !log ori synchronized php-1.23wmf10/extensions/WikimediaEvents [01:19:13] Logged the message, Master [01:26:48] !log bsitu synchronized php-1.23wmf11/extensions/Flow 'Update Flow with some special contribs cherry-picks' [01:26:56] Logged the message, Master [01:31:42] So we deployed some fixes to Flow changes in Special:Contributions to mediawiki.org. [01:36:51] <^d> !log gerrit down for upgrade [01:37:00] Logged the message, Master [01:39:43] PROBLEM - gerrit process on ytterbium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^GerritCodeReview .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war [01:39:50] gj icinga-wm [01:40:18] you can disable alerts in icinga-admin.wikimedia.org , though it may require ops [01:40:55] I suspect it wont be as bad as the puppet spam ;) [01:42:04] <^d> It'll be back soon enough. [01:42:10] <^d> database already upgraded. [01:42:21] <^d> re-running puppet now, then will bring service back up [01:42:43] RECOVERY - gerrit process on ytterbium is OK: PROCS OK: 1 process with regex args ^GerritCodeReview .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war [01:43:09] <^d> !log gerrit back up, running v2.8.1 stable now [01:43:19] Logged the message, Master [01:43:57] <^d> spage: Definitely has the old change screen as the default. You can turn the ugly thing on if you'd like. [01:44:03] <^d> (although I don't know why you would :)) [02:20:23] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:45] !log LocalisationUpdate completed (1.23wmf10) at 2014-01-28 02:25:44+00:00 [02:25:53] PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:25] andrewbogott: ^ [02:29:03] lol, wikitech [02:44:52] (03PS1) 10Springle: prepare for rotation of s6 master from db1027 to db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109824 [02:47:06] virt0 /a full [02:47:27] dejavu [02:48:35] yeah [02:48:45] * springle deletes two oldest backups [02:49:43] RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 4.237 second response time [02:50:02] springle: Thanks! [02:52:03] !log LocalisationUpdate completed (1.23wmf11) at 2014-01-28 02:52:02+00:00 [02:52:15] Logged the message, Master [02:53:13] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.421 second response time [02:53:34] !log wikitech /a full again as per Ryan Lane email to ops@ on 2014-01-14. Deleted two oldest backup sets *2014012[01]*. [02:53:42] Logged the message, Master [02:54:54] (03CR) 10Springle: [C: 032] prepare for rotation of s6 master from db1027 to db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109824 (owner: 10Springle) [02:55:38] scfc_de: back, looking... [02:55:45] oh, and now it's recovered [02:55:46] Sean just fixed it ;) [02:55:57] well it isn't fixed [02:56:00] springle: disk full? [02:56:01] it needs attention [02:56:10] yes. backups keep overflowing [02:56:15] ok [02:56:17] You fixed wikitech [02:57:54] !log springle synchronized wmf-config/db-eqiad.php 'db1006 to LB 400. prep db1027 for s6 master rotation' [02:58:01] Logged the message, Master [03:09:19] (03PS1) 10Springle: rotate s6 master, demote db1027, promote db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109828 [03:09:43] PROBLEM - mysqld processes on db1027 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [03:09:57] (03CR) 10Springle: [C: 032] rotate s6 master, demote db1027, promote db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109828 (owner: 10Springle) [03:10:04] (03Merged) 10jenkins-bot: rotate s6 master, demote db1027, promote db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109828 (owner: 10Springle) [03:11:14] !log springle synchronized wmf-config/db-eqiad.php 'rotate s6 master, demote db1027, promote db1023' [03:11:21] Logged the message, Master [03:11:43] PROBLEM - MySQL Replication Heartbeat on db1015 is CRITICAL: CRIT replication delay 307 seconds [03:11:43] PROBLEM - MySQL Replication Heartbeat on db1006 is CRITICAL: CRIT replication delay 307 seconds [03:11:45] !log springle synchronized wmf-config/db-pmtpa.php 'rotate s6 master, demote db1027, promote db1023' [03:11:52] (03CR) 10Andrew Bogott: [C: 032] set time intervals properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/109701 (owner: 10Cmcmahon) [03:11:52] Logged the message, Master [03:11:53] PROBLEM - MySQL Replication Heartbeat on db1010 is CRITICAL: CRIT replication delay 321 seconds [03:11:55] say what now [03:12:13] PROBLEM - MySQL Replication Heartbeat on db1022 is CRITICAL: CRIT replication delay 334 seconds [03:12:33] PROBLEM - MySQL Replication Heartbeat on db1023 is CRITICAL: CRIT replication delay 355 seconds [03:13:31] false positive [03:13:38] * springle phew [03:20:45] springle, the full volume on vir0 was /a? [03:20:55] andrewbogott: correct [03:23:08] (03PS1) 10Springle: update coredb topology after s6 master rotation [operations/puppet] - 10https://gerrit.wikimedia.org/r/109831 [03:23:08] crontab has [03:23:08] # Puppet Name: backup-cleanup [03:23:09] 0 3 * * * find /a/backup -type f -mtime +7 -delete [03:23:18] springle, maybe I should just change that to +4 [03:23:30] is 4 enough? [03:23:30] Or does that strike you as an embrassing half-measure? [03:23:34] who decides that [03:23:58] well… I don't really know what that backup is for anyway. Guess I should figure that out [03:24:28] Ah, it's backups of all the openstack bits [03:24:29] (03CR) 10Springle: [C: 032] update coredb topology after s6 master rotation [operations/puppet] - 10https://gerrit.wikimedia.org/r/109831 (owner: 10Springle) [03:29:18] (03PS1) 10Andrew Bogott: Save backups for 4 days rather than 7. [operations/puppet] - 10https://gerrit.wikimedia.org/r/109833 [03:32:00] (03CR) 10Andrew Bogott: [C: 032] "Better than an outage." [operations/puppet] - 10https://gerrit.wikimedia.org/r/109833 (owner: 10Andrew Bogott) [03:32:36] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-01-28 03:32:36+00:00 [03:32:45] Logged the message, Master [03:40:50] (03PS1) 10Ori.livneh: retire logmsgbot from #wikimedia-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/109836 [03:41:06] (03PS2) 10Ori.livneh: retire logmsgbot from #wikimedia-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/109836 [03:41:19] (03CR) 10Ori.livneh: [C: 032 V: 032] retire logmsgbot from #wikimedia-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/109836 (owner: 10Ori.livneh) [03:42:14] just needs a puppet run on neon now [03:51:38] (03PS1) 10Yurik: Added an opera-only partner [operations/puppet] - 10https://gerrit.wikimedia.org/r/109838 [03:52:16] (03PS1) 10Springle: update cname after s6 master rotation [operations/dns] - 10https://gerrit.wikimedia.org/r/109839 [03:52:41] (03PS1) 10Ori.livneh: tcpircbot: allow 'channels' to be either a string or an array [operations/puppet] - 10https://gerrit.wikimedia.org/r/109840 [03:52:58] (03CR) 10Springle: [C: 032] update cname after s6 master rotation [operations/dns] - 10https://gerrit.wikimedia.org/r/109839 (owner: 10Springle) [03:54:54] RECOVERY - MySQL Replication Heartbeat on db1010 is OK: OK replication delay -0 seconds [03:55:04] RECOVERY - MySQL Replication Heartbeat on db1022 is OK: OK replication delay -0 seconds [03:55:34] RECOVERY - MySQL Replication Heartbeat on db1023 is OK: OK replication delay -1 seconds [03:55:54] RECOVERY - MySQL Replication Heartbeat on db1015 is OK: OK replication delay -0 seconds [03:55:54] RECOVERY - MySQL Replication Heartbeat on db1006 is OK: OK replication delay -0 seconds [03:56:20] (03PS2) 10Ori.livneh: tcpircbot: allow 'channels' to be either a string or an array [operations/puppet] - 10https://gerrit.wikimedia.org/r/109840 [03:58:25] (03PS3) 10Ori.livneh: tcpircbot: allow 'channels' to be either a string or an array [operations/puppet] - 10https://gerrit.wikimedia.org/r/109840 [03:58:34] (03PS4) 10Ori.livneh: tcpircbot: allow 'channels' to be either a string or an array [operations/puppet] - 10https://gerrit.wikimedia.org/r/109840 [03:58:40] (03CR) 10Ori.livneh: [C: 032 V: 032] tcpircbot: allow 'channels' to be either a string or an array [operations/puppet] - 10https://gerrit.wikimedia.org/r/109840 (owner: 10Ori.livneh) [04:17:02] (03PS1) 10Springle: depool db1040 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109842 [04:17:20] (03CR) 10Springle: [C: 032] depool db1040 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109842 (owner: 10Springle) [04:17:26] (03Merged) 10jenkins-bot: depool db1040 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109842 (owner: 10Springle) [04:18:23] !log springle synchronized wmf-config/db-pmtpa.php 'depool db1040 for schema changes' [04:18:30] Logged the message, Master [04:18:56] !log springle synchronized wmf-config/db-eqiad.php 'depool db1040 for schema changes' [04:19:03] Logged the message, Master [04:58:44] RobH: I'm having trouble accessing https://wikitech with curl/wget and andrewbogott said you're working on a "proper" certificate (apparently https://rt.wikimedia.org/Ticket/Display.html?id=6592). Do you have an ETA when that will be available? [05:40:43] ottomata1: so [05:40:54] ottomata1: https://github.com/trebuchet-deploy [05:41:06] I've been slowly working everything into the upstream [05:41:40] trebuchet itself is still in kind of a grey zone of being correctly up to date in puppet and not on github [05:42:03] Gloria: yeah re jan 20th, just didn't have any time to archive it today, will do [05:42:15] or, "it's a wiki" ;) [05:42:27] :D [05:42:28] ottomata1: you'll find the trebuchet code in puppet in the "deployment" module [05:42:41] greg-g: Do you know if Flow has been rescheduled? [05:44:31] not yet [05:44:43] ottomata1: I'll try to get that code properly in github and in launchpad ppas [05:59:17] Man, when I see that a puppet module has all its logic in .rb rather than .pp I'm immediately suspicious. [05:59:30] Ryan_Lane: Speaking of puppet-nova here :/ [07:06:28] (03PS1) 10Ori.livneh: Add VisualEditor timing data reporter [operations/puppet] - 10https://gerrit.wikimedia.org/r/109848 [07:43:41] (03PS1) 10Matanya: emery: remove one udp2log logger. [operations/puppet] - 10https://gerrit.wikimedia.org/r/109849 [08:07:15] (03CR) 10Ori.livneh: [C: 032] Add VisualEditor timing data reporter [operations/puppet] - 10https://gerrit.wikimedia.org/r/109848 (owner: 10Ori.livneh) [08:22:14] (03PS4) 10Yuvipanda: Deploy Extension:MobileApp to betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106217 [08:22:34] (03CR) 10Hashar: [C: 032] Deploy Extension:MobileApp to betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106217 (owner: 10Yuvipanda) [08:22:42] (03Merged) 10jenkins-bot: Deploy Extension:MobileApp to betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106217 (owner: 10Yuvipanda) [08:22:47] hashar: wooo! [09:01:16] (03CR) 10Alexandros Kosiaris: "@ottomata" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109316 (owner: 10Ottomata) [09:16:52] (03CR) 10Faidon Liambotis: "Why a submodule?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109316 (owner: 10Ottomata) [09:17:42] paravoid: because it is also being used in Mediawiki-Vagrant, IIRC [09:17:51] (03CR) 10Faidon Liambotis: [C: 04-1] "w" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109838 (owner: 10Yurik) [09:22:57] (03CR) 10Dzahn: [C: 032] "icinga host group was erroneously still named "pmtpa search servers (lucene)" but now only contains eqiad servers" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109635 (owner: 10Dzahn) [09:25:29] (03CR) 10Alexandros Kosiaris: [C: 032] network: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109732 (owner: 10Matanya) [09:35:39] (03PS2) 10Yurik: Zero: Added 655-12 [operations/puppet] - 10https://gerrit.wikimedia.org/r/109838 [09:36:37] andrewbogott: hi, i didn't really understand whey you merged the cron change (the first one) yesterday, did i miss something? [09:37:13] matanya: Um… I merged without thinking that hard about it/figuring it was mostly harmless [09:37:19] That's correct now in any case, right? [09:37:37] (And, actually, I tried to get puppet-lint to complain about the lack of quotes and it didn't. But maybe a problem with my methodology.) [09:37:51] yes, i just thought you saw something i missed [09:38:20] regarding quotes, not lint warning, just a style thing I (we?) do [09:38:46] I should write a draft style guide, shouldn't i? [09:39:45] https://wikitech.wikimedia.org/wiki/Puppet_coding#Coding_Style isn't enough i fear [09:41:36] akosiaris: i would much appricate if you can review my site lint patch. i know it is a touchy one [09:42:47] matanya, if your style guide isn't too long you can shoehorn it into the 'Format' section... [09:42:57] matanya: yeah i 'll try [09:43:28] andrewbogott: i need to get OPS ok before deciding such thing on my own [09:43:34] thanks akosiaris [09:43:45] matanya: true, you can write a draft someplace else and we can discuss adding it to that page [09:43:54] ok [09:45:23] (03PS3) 10Faidon Liambotis: Zero: add carrier 655-12 to Varnish config [operations/puppet] - 10https://gerrit.wikimedia.org/r/109838 (owner: 10Yurik) [09:45:34] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Zero: add carrier 655-12 to Varnish config [operations/puppet] - 10https://gerrit.wikimedia.org/r/109838 (owner: 10Yurik) [09:48:32] (03CR) 10Faidon Liambotis: [C: 04-1] coredb_mysql: puppet 3 compatibility fix: fully qualify variable (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108313 (owner: 10Matanya) [09:50:12] (03CR) 10Faidon Liambotis: [C: 04-1] coredb_mysql: puppet 3 compatibility fix: fully qualify variables (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108488 (owner: 10Matanya) [09:51:02] (03CR) 10Faidon Liambotis: "Please do not rebase needelessly. It doesn't make a difference for review purposes, and we can always rebase right before the merge." [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 (owner: 10Physikerwelt) [09:52:14] (03PS2) 10Matanya: coredb_mysql: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/108313 [09:52:24] (03PS3) 10Faidon Liambotis: ldap: puppet 3 compatibility, fix variable name typo [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 (owner: 10Matanya) [09:52:34] (03CR) 10Faidon Liambotis: [C: 032] ldap: puppet 3 compatibility, fix variable name typo [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 (owner: 10Matanya) [09:52:36] (03CR) 10jenkins-bot: [V: 04-1] ldap: puppet 3 compatibility, fix variable name typo [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 (owner: 10Matanya) [09:52:54] pfft, needs rebase [09:53:10] doing paravoid [09:53:19] (03PS2) 10Dzahn: remove professor from site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/109285 [09:53:54] mutante: feel free to proceed with that without CR, as long as you follow the decom process :) [09:54:44] (03CR) 10ArielGlenn: [C: 032] remove professor from site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/109285 (owner: 10Dzahn) [09:54:45] paravoid: better do the change in another patch, my ldap-lint stuff changed it a lot [09:54:57] paravoid: yea, thanks for yours and ori's trust, be i just talked to apergos about the "follow the decom process"-part, especially the right order and the non-gerrit part like cleaning certs, salt keys [09:55:03] matanya: keep the same Change-Id [09:55:07] i will [09:55:09] matanya: and it will get under the same gerrit patchset [09:57:42] (03PS4) 10Matanya: ldap: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 [09:57:49] done paravoid [09:58:09] no, not good [09:58:28] (03CR) 10jenkins-bot: [V: 04-1] ldap: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 (owner: 10Matanya) [09:59:02] (03PS5) 10Matanya: ldap: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 [09:59:36] (03CR) 10Faidon Liambotis: [C: 032] ldap: puppet 3 compatibility fix: fully qualify variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/107823 (owner: 10Matanya) [10:00:11] (03PS1) 10Andrew Bogott: ::qualify the global openstack_version [operations/puppet] - 10https://gerrit.wikimedia.org/r/109858 [10:00:46] (03CR) 10jenkins-bot: [V: 04-1] ::qualify the global openstack_version [operations/puppet] - 10https://gerrit.wikimedia.org/r/109858 (owner: 10Andrew Bogott) [10:01:50] andrewbogott: and abandon https://gerrit.wikimedia.org/r/#/c/97007 presumably? [10:02:21] Oh, I don't know, maybe that one's better! [10:02:23] * andrewbogott reads [10:05:44] paravoid, apergos, I pretty much don't understand about the class vs. global question in that change. In the style guide we have 'Role classes are never parameterized, and are only configured via globals' which I wrote and maybe didn't think about enough... [10:06:07] Does it not work to have site.pp define $foo and then refer to it later as $::foo? [10:06:34] not if it'sn in a node definition, that'snot top level [10:07:55] So the whole idea of configuring a role via a global… impossible? Or just ugly due to ambiguous scope? [10:08:15] Wrapping each var in a class seems like a whole lot of work [10:08:26] (and can't really be done from labsconsole either atm) [10:08:48] paravoid: do you want to take https://gerrit.wikimedia.org/r/#/c/83768/ until we made some progress on a more general approach in https://gerrit.wikimedia.org/r/#/c/107831/ , that might take a bit longer though, if the simpler one is bad though i'll abandon it in favor of the other one [10:08:50] yeah I don't know what a good approach is [10:09:03] the second one was kind of a reply to your comment on the first one [10:09:06] but the dynamic scope thing means you can't do it that way in puppet3 [10:09:30] (03CR) 10Faidon Liambotis: [C: 032] move {jobs,careers}.wikimedia.org to redirects.dat [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106108 (owner: 10Jeremyb) [10:11:12] (03CR) 10Faidon Liambotis: [C: 04-1] "I'd prefer the opposite: trailing slashes everywhere." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106110 (owner: 10Jeremyb) [10:11:35] I was just looking for some way to avoid dynamic scope, there must be better ones [10:11:42] apergos: Tell me more about 'the dynamic scope thing'? Does that mean no globals ever? [10:12:08] you have to give the full reference to the var name [10:12:59] e.g. $::foo [10:13:02] is that a full reference? [10:13:08] yes but you can't assign to it [10:13:17] ariables can only be assigned using their short name [10:13:21] *variables [10:13:28] you can retrieve from it [10:13:49] ok, so then what's an example of assigning and subsequently retrieving? [10:14:42] if you are in role/foo.pp and you want it to use class also called foo, but which is from modules/foo/manifests/init.pp (which has to have that name by module convention), then you'd have to use it like class { '::foo': parameter => in the role class [10:14:52] Or is the answer to that 'it can only be done via parameters'? [10:14:57] common mistake was to forget the leading :: and then the role class tries to find itself [10:15:55] (03CR) 10Faidon Liambotis: [C: 04-1] "This is very nice work, kudos for doing all this!" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106109 (owner: 10Jeremyb) [10:17:12] (03Abandoned) 10Hashar: cleanup scap scripts, sql, etc. [operations/puppet] - 10https://gerrit.wikimedia.org/r/8438 (owner: 10Jeremyb) [10:17:34] in search.pp lucene::server has an indexer value passed in [10:17:37] oh shit [10:17:38] (03PS1) 10Dzahn: remove professor from dsh groups (decom) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109860 [10:17:42] later in the config class ther eis a line if $lucene::server::indexer == true [10:17:47] that's pretty typical [10:17:52] (for an example) [10:19:02] (03Abandoned) 10Hashar: db1038 is s3 master [operations/puppet] - 10https://gerrit.wikimedia.org/r/90481 (owner: 10Springle) [10:19:17] (03Abandoned) 10Hashar: db1027 is s6 master [operations/puppet] - 10https://gerrit.wikimedia.org/r/89616 (owner: 10Springle) [10:19:31] paravoid: ? [10:19:35] (03CR) 10Dzahn: [C: 032] "per https://wikitech.wikimedia.org/wiki/Server_Lifecycle#In_Service :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109860 (owner: 10Dzahn) [10:19:50] (03Abandoned) 10Hashar: Inserted QT libraries [operations/puppet] - 10https://gerrit.wikimedia.org/r/96478 (owner: 10Petrb) [10:19:53] nothing [10:19:55] false alarm [10:19:58] :) ok [10:20:40] apergos: it isn't passed in though, right? It's globally defined? [10:20:43] Just as a class vs. a var? [10:21:11] (03Abandoned) 10Hashar: more permissions/ownership hackery [operations/puppet] - 10https://gerrit.wikimedia.org/r/58614 (owner: 10Jgreen) [10:21:20] oh, you want an example of a global var assigned and then later retrieved? sorry, I lost track a little of what you need [10:21:33] apergos: eh, request for command line to add to docs, if you have it handy [10:21:36] Remove from puppet stored configuration files. [10:21:51] i know i did it before, looks in history [10:22:07] apergos, I think you are telling me that https://wikitech.wikimedia.org/wiki/Puppet_coding#Organization is mostly wrong and/or impossible. [10:22:18] Which means I am immobilized, pending new instructions :) [10:22:25] (03Abandoned) 10Hashar: fundraising.pp file owner cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/58603 (owner: 10Jgreen) [10:22:40] it's on the server lifecycle page [10:22:44] The thing you do for the above OpenStack patch works but I don't see how it's better than just using a parameter in the first place [10:22:59] Manually run puppetstoredconfigclean.rb on the puppet master. [10:23:01] mutante: [10:23:03] …which, neither thing is supported by labs, which makes me sad :( [10:23:09] andrewbogott: looking [10:23:35] 1 on there is certainly wrong for moving to puppet3 [10:23:49] apergos: thanks, that was it, just needed reminder of the script name [10:24:01] (adds in that place on lifecycle page) [10:24:08] mutante: it's already there [10:24:33] the line I pasted in here, it's from that page under the right step [10:24:33] apergos: well, it says to " [10:24:33] Remove from puppet stored configuration files. [10:24:38] (03Abandoned) 10Hashar: Send VisualEditor metrics to Ganglia via StatsD [operations/puppet] - 10https://gerrit.wikimedia.org/r/90464 (owner: 10Ori.livneh) [10:24:38] that's a bad title [10:24:41] as well _before_ that warning [10:24:45] that means from the manifests [10:24:48] These steps, once started, must be completed without interruption. [10:25:12] apergos: duh, then i just read that part as meaning the storedconfigclean.rb [10:25:18] apergos: ok, got it, no worries [10:25:36] if you want to change that section title so it's clearer that's fine [10:25:47] ok [10:28:49] !log removed professor from dsh,puppet,puppetmaster: on palladium Killing professor.pmtpa.wmnet...done. [10:28:52] apergos: this isn't urgent since I can just merge your patch as is. But will you take on the task of writing some docs (or at least opening a discussion) about how we should properly define the boundaries between nodes, roles, and modules? [10:28:58] Logged the message, Master [10:29:08] Since my attempt won't work with 3? [10:29:39] well but the patch is considered the wrong approach [10:29:53] so please don't merge it :-( [10:30:00] ok... [10:30:09] the thing is I don't know what a good approach is [10:30:13] I guess my patch was just clarifying anyway, I can just strip out everything but the comments [10:30:26] I suspect that the right approach is 'use heira' [10:30:28] I am happy to open a discussion, where were you thinking? [10:30:32] but I don't even really know what heira is. [10:30:49] apergos, on wikitech-l probably. [10:31:02] ryan has ben using it at $newjob [10:31:05] Unless you have a complete fully-formed vision in which case you can just edit the wiki by fiat [10:31:22] no, I have a lack of vision [10:31:34] ok I'll send an email in a bit [10:31:36] My instinct about heira is -- I like being able to read about things in files, like having to look inside a db less. [10:31:43] But I may be misunderstanding how it works. [10:31:44] Thanks. [10:33:55] (03PS5) 10Matanya: site: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109507 [10:34:16] I should read up [10:34:18] (03Abandoned) 10Andrew Bogott: ::qualify the global openstack_version [operations/puppet] - 10https://gerrit.wikimedia.org/r/109858 (owner: 10Andrew Bogott) [10:34:48] !log disabled host and service checks/notifications for professor. running puppet on icinga [10:34:55] Logged the message, Master [10:35:10] andrewbogott: heira would be a good approach here [10:35:35] you will just being assigning the "data" in heira a variable [10:36:22] apergos: heh, disabled notifications just in time, icinga config didnt break but is now "host not found" a minute after i ACKed the services [10:36:29] that is a good abstraction layer, and you can chage the data held in heira as much as you want as long as the variable pointing to it remains the same [10:36:42] matanya: Is that different from a global? [10:37:08] yes, it is held "outside" of puppet scope [10:37:25] this prevents scoping issues [10:37:34] I understand that it is technically different from a global, but from a code-design standpoint is it the moral equivalent of a global? [10:37:42] Just whitewashed by the purity of 'not being code'? [10:38:11] mostly similar, but has some diff's [10:38:47] and is great when using stuff like passwords [10:39:33] (03CR) 10JanZerebecki: coredb_mysql: puppet 3 compatibility fix: fully qualify variables (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108488 (owner: 10Matanya) [10:40:28] matanya: makes sense for passwords, right now the line we draw between public and private puppet info is funny [10:41:26] !log professor: revoked puppet cert, deleted salt minion key, powering down. bye bye [10:41:34] Logged the message, Master [10:43:38] (03CR) 10Faidon Liambotis: coredb_mysql: puppet 3 compatibility fix: fully qualify variables (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108488 (owner: 10Matanya) [10:44:27] (03CR) 10Hashar: [C: 04-1] "Well funnel will redirect anything under those domains to one and only URL. So that would break any potential old links. We used to have " [operations/apache-config] - 10https://gerrit.wikimedia.org/r/109652 (owner: 10Dzahn) [10:44:44] mutante: I guess you can get rid of the bugzilla funnel :D https://gerrit.wikimedia.org/r/#/c/109652/ [10:44:57] (03CR) 10Matanya: coredb_mysql: puppet 3 compatibility fix: fully qualify variables (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108488 (owner: 10Matanya) [10:45:18] mutante: the SSL error for https://bugzilla.wikipedia.org/ should be fixed by pointing it to the mediawiki load balancer instead of pointing to kaulen [10:46:08] hashar: confusing, because that's exactly what i said on that ticket in Oct 2013, but then [10:46:11] that's fixed meanwhile, apparently: [10:46:12] wikipedia.org zone: 73 bugs 1H IN CNAME bugs.wikimedia.org. [10:46:35] let me look again at your comments first [10:47:26] hashar: i did that because i looked at your example from doc.mediawiki.org to doc.wikimedia.org and there is no cert error [10:47:37] or at least very similar thing [10:48:12] mutante: yeah there is no cert because that points to the text load balancer and make the SSL query to terminate on the SSL proxy which have the *.mediawiki.org cert [10:48:19] where do you see that pointed to kaulen? [10:48:29] i just see it in wikiMedia zone [10:48:50] apergos: a wikitech post for a simple puppet question? seriously? [10:48:55] bugzilla.wikipedia.org. 3600 IN CNAME bugzilla.wikimedia.org. [10:48:58] jesus [10:49:12] andrewbogott: skim over: http://www.slideshare.net/PuppetLabs/roles-talk [10:49:18] for the single puppet3 commit you've done? [10:49:34] thankfully matanya isn't writing a wikitech-l mail for every puppet3 fix they are making [10:49:56] mutante: so I guess you want to move bugzilla.wikipedia.org to text load balancer, making sure the apaches have it as a server alias [10:50:03] paravoid: I asked him to write that post, because it's an important question. We have a style guide which is almost entirely incorrect for puppet 3. [10:50:17] I'd like to start writing puppet 3-appropriate code as soon as possible. [10:50:37] paravoid: i can start now :) [10:50:47] what is it specifically on our style guide that is incorrect for puppet 3? [10:50:55] hashar: i'm confused, that's what the ticket started out as and apache redirects have that alias since a longer time [10:51:13] paravoid: https://wikitech.wikimedia.org/wiki/Puppet_coding#Organization [10:51:13] but i'm also multi-tasking, so i'll look closer in a sec [10:51:35] The interface between nodes, roles, manifests. This also matters to how the labs console handles puppet [10:51:38] since right now it relies on globals [10:51:46] matanya: please don't :) your amazing in volume work and lack of procrastination is highly appreciated. many kudos. [10:51:55] andrewbogott: what specifically? [10:52:04] can you point me to the relevant section? [10:52:23] That is a specific section already :) Items 1,2 and 3 all mention 'globals' and none of those things will work in 3. [10:52:28] As I understand it. [10:52:28] mutante: the text apaches are probably already configured (haven't checked), you still need to point the DNS entry to them :-D Currently bugzilla.wikipedia.org points to kaulen which does not have the SSL cert for it (nor for *.wikipedia.org). [10:52:51] thanks paravoid :) [10:53:04] hashar: https://gerrit.wikimedia.org/r/#/c/108906/ [10:53:22] andrewbogott: I don't understand... [10:53:44] what exactly is the problem? could you give an example? [10:53:59] paravoid: ok… I think that this whole conversation happened here in this room, in response to your comment :) So [10:54:55] So my understanding is that declaring $foo=7 in site.pp and then referring to $foo in a subsequent class won't work due to how scope is handled in puppet 3. [10:55:02] that is correct [10:55:25] So, the question is, how to get node-specific config into a role class? [10:55:36] One answer is, 'make role class parameterized' but we don't do that currently, on purpose. [10:55:39] like what? [10:55:48] There are, presumably, other answers. [10:56:01] (we do that, for coredb, but it's an exception) [10:56:06] (we generally try to avoid doing so) [10:56:17] that should be removed [10:56:30] paravoid: if you are indirectly arguing that role classes should always be self-sufficient and not take params from outside, well, that would be a valid response to this question. [10:56:50] for openstack_version, the right way to do it would be to have a role class called "role::labs::eqiad" or something like that [10:57:06] or role::labs::node, that has a conditional on $::site within it [10:57:20] the latter being better [10:57:29] role classes exist specifically for /not/ being generic [10:57:31] So, ok, you are advocating for a possible answer to this question. And I support the advocating of answers! [10:57:41] like for example, we have role::authdns::ns0, role::authdns::ns1, role::authdns::ns2 [10:57:47] but you having an answer is not the same thing as thinking the question is dumb or trivial. [10:57:56] There are plenty of vars defined in site.pp currently. [10:58:04] And labsconsole allows the configuration of instances that way [10:58:08] yup, we're working on fixing this [10:58:08] And the style guide encourages it. [10:58:14] where does it encourage it? [10:58:31] Use of global variables within non-role manifests or templates is highly discouraged. A few actually global settings (e.g. $::site, $::realm) will crop up, but we probably don't need any new ones. [10:58:52] So… I wrote that. [10:58:56] non-role manifests. [10:59:01] good :) [10:59:04] As opposed to role manifests, which 4. Role classes are never parameterized, and are only configured via globals. [10:59:17] and 1. Nodes defined in site.pp can include role classes and define global variables. [10:59:23] and I didn't say the question is dumb, fwiw [10:59:35] ok, sorry [11:00:10] that said, i do think we should move as much 'data' out of code and use hiera. [11:00:21] Anyway -- if you have a definite, right answer for how this should be managed, then, that's great! Apergos and I just now had a long discussion about this and neither of us did. [11:00:38] using heira strikes me as a different solution from 'fully self-sufficient roles', yes? [11:00:46] apergos has largely ignored the puppet3 effort for months now, and has a single commit about it, and takes it up on wikitech-l to discuss that single one commit, while matanya for example has been doing all of https://etherpad.wikimedia.org/p/Puppet3 without too much fuss [11:00:53] this is what annoys me, to be clear. [11:01:26] (while apergos had commited to help with the puppet3 migration many weeks ago) [11:01:34] to be clear, I asked multiple times, every few days, someone (I am not going to name names because this is not about blaming them) for a review of that one change, so that I could knowo if that was the right approach before proceeding [11:01:38] I never got an answer [11:01:52] that went on for several weeks. [11:01:59] Yeah, and we just now discussed it at lenght in this channel and no one piped in with an answer... [11:02:03] Mailing list seems like the right next step [11:02:08] that is not the same as ignoring the effort. [11:02:39] apergos: so you started with a single commit from a non-trivial case, got no response to reviews, and decided to stop altogether? [11:02:53] when you weighed in, you told me basically to leave it alone and let other people refactor it, but that still did not tell me what the right approach is, which I asked you then too [11:02:59] I'm sorry, but https://gerrit.wikimedia.org/r/#/q/owner:%22ArielGlenn+%253Cariel%2540wikimedia.org%253E%22,n,z vs. https://gerrit.wikimedia.org/r/#/q/owner:%22Matanya+%253Cmatanya%2540foss.co.il%253E%22,n,z [11:03:06] something is wrong in this picture, isn't it? [11:03:07] from that I get the very clear conclusion: stay the heck out [11:03:32] aside from who did or didn't do something, can we look at the right approach? [11:04:12]