[00:50:42] !help Hi there, I'm looking to host a bot on the Tool Labs - would it be possible to get an approved membership? Cheers [00:50:42] kolbert: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [01:03:06] kolbert: {{done}}. Don't forget to check the local bot policies on the wikis where you want to run your bot. :) [01:03:22] and welcome to Toolforge [01:03:23] thank you :) [01:08:45] (03PS1) 10BryanDavis: Add privacy warnings to account creation [labs/striker] - 10https://gerrit.wikimedia.org/r/373169 (https://phabricator.wikimedia.org/T173845) [01:24:43] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:25:38] thanks bd808 [01:29:42] (03CR) 10Andrew Bogott: Add privacy warnings to account creation (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/373169 (https://phabricator.wikimedia.org/T173845) (owner: 10BryanDavis) [02:04:45] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [02:26:04] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:29:17] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [02:31:29] !help Hello! I was just given production shell access (https://phabricator.wikimedia.org/T171988) and am having a bit of trouble connecting. [02:31:29] Shilad: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [02:32:13] I'm wondering, first of all, if my password should be the same as wikitech / gerritt. I presume the answer is yes? [02:32:17] Shilad: What are you seeing? And what OS are you using locally? [02:32:45] Passwords shouldn't be involved at all… let me find you a link [02:33:01] I'm use OSX, and I'm seeing it ask me for a password. Maybe my key isn't being configured properly... [02:33:23] yeah, if you get a password prompt it's because the key exchange is failing. [02:33:29] What system are you connecting to? [02:34:19] ssh -vv bast1001.wikimedia.org shows this suspicious line: [02:34:27] debug1: Skipping ssh-dss key /Users/a558989/.ssh/wmf_prod - not in PubkeyAcceptedKeyTypes [02:35:07] Are DSA keys allowed? [02:35:16] I think, yes, dsa will work and dss will not. [02:36:00] That file begins with "-----BEGIN DSA PRIVATE KEY-----" so I presume it is, in fact, a DSA key. [02:36:48] Are you following a guide on wikitech someplace? e.g. https://www.mediawiki.org/wiki/Gerrit/Tutorial#Set_Up_SSH_Keys_in_Gerrit ? [02:37:05] But the log line does make it look like a DSS key. Weird. Maybe that's the problem... [02:37:30] This is where I would start, if you're not there already: https://wikitech.wikimedia.org/wiki/Help:Access [02:38:23] This warning appears in that page "This is not about Wikimedia production cluster access, for that, please see production shell access." [02:38:46] Are there some parts that are still applicable? [02:38:48] oooh [02:38:54] wait, what are you actually trying to do? [02:39:04] You're in the -cloud channel so I assumed you were trying to reach a cloud instance [02:39:15] but bast1001.wikimedia.org is not [02:39:51] Sorry! Is there an IRC for the production cluster? I could also follow up on phab if that's better. [02:39:59] This isn't time critical. I was just eager :) [02:41:27] And I was following this guide: https://docs.google.com/document/d/1BwB92e-wNc-y6c5DYfBj7ZxdRFmYlKa-ijzp4t-2f0c/edit [02:43:12] typically you'd want to ask in wikimedia-operations if it's about production hosts. [02:43:18] I'll meet you there :) [02:44:43] OMW [02:50:45] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [03:01:05] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [03:04:17] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [03:30:45] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [03:39:34] PROBLEM - Puppet errors on tools-exec-1429 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:04:35] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [04:39:39] (03PS2) 10BryanDavis: Add privacy warnings to account creation [labs/striker] - 10https://gerrit.wikimedia.org/r/373169 (https://phabricator.wikimedia.org/T173845) [04:39:52] (03CR) 10BryanDavis: Add privacy warnings to account creation (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/373169 (https://phabricator.wikimedia.org/T173845) (owner: 10BryanDavis) [05:28:37] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:03:38] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [06:04:15] (03PS1) 10Legoktm: Generate reports for sniffs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/373179 [06:04:17] (03PS1) 10Legoktm: Empty sniff directory before writing new ones [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/373180 [06:11:51] (03CR) 10Legoktm: [C: 032] Generate reports for sniffs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/373179 (owner: 10Legoktm) [06:11:54] (03CR) 10Legoktm: [C: 032] Empty sniff directory before writing new ones [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/373180 (owner: 10Legoktm) [06:12:10] (03Merged) 10jenkins-bot: Generate reports for sniffs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/373179 (owner: 10Legoktm) [06:12:12] (03Merged) 10jenkins-bot: Empty sniff directory before writing new ones [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/373180 (owner: 10Legoktm) [06:42:53] PROBLEM - Puppet errors on tools-grid-master is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:44:40] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:17:24] 10Toolforge: Cannot log into Tool Labs local DB via 'sql' command - https://phabricator.wikimedia.org/T173708#3543762 (10Magnus) 05Open>03Resolved a:03Magnus Everything works now after renaming/removing of .my.cnf Thanks! [07:17:54] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:41] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:59] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:10:58] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [08:48:25] !log tools.stewardbots Added Melos +2 rights on Gerrit for our repo. [08:48:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [09:41:01] 10Tool-wikiloves: Bug on a page of http://tools.wmflabs.org/wikiloves - https://phabricator.wikimedia.org/T173583#3543995 (10JeanFred) It looks like it’s working now... http://tools.wmflabs.org/wikiloves/monuments/2016/France [13:34:34] 10Tool-fatameh: Help users respect maxlag - https://phabricator.wikimedia.org/T173921#3544470 (10Tarrow) [14:10:58] 10Tool-fatameh: Help users respect maxlag - https://phabricator.wikimedia.org/T173921#3544672 (10Tarrow) Ticket for WikidataIntegrator is https://github.com/SuLab/WikidataIntegrator/issues/34 [14:27:58] Technical Advice IRC meeting starting at 3 pm UTC/5 pm MEZ in channel #wikimedia-tech, hosts: @addshore & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:06:47] (03CR) 10Andrew Bogott: [C: 032] Add privacy warnings to account creation [labs/striker] - 10https://gerrit.wikimedia.org/r/373169 (https://phabricator.wikimedia.org/T173845) (owner: 10BryanDavis) [15:08:04] (03Merged) 10jenkins-bot: Add privacy warnings to account creation [labs/striker] - 10https://gerrit.wikimedia.org/r/373169 (https://phabricator.wikimedia.org/T173845) (owner: 10BryanDavis) [15:08:18] andrewbogott: thanks :) [15:23:48] (03PS5) 10BryanDavis: Tool account creation and more [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/370139 (https://phabricator.wikimedia.org/T128400) [15:29:51] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:32:38] (03CR) 10BryanDavis: [C: 032] Tool account creation and more [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/370139 (https://phabricator.wikimedia.org/T128400) (owner: 10BryanDavis) [15:32:44] (03Merged) 10jenkins-bot: Tool account creation and more [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/370139 (https://phabricator.wikimedia.org/T128400) (owner: 10BryanDavis) [16:09:53] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:09] bd808: What is this rule for in tools.versions? ".wikis li a:hover:after {background:#f0f0f0;bottom:0;content:attr(id);left:0;padding-right:.5em;position:absolute;z-index:98;}" [16:14:28] It seems to replace the text content with an ::after overlay of the same content, except 1px lower [16:15:13] it shows things that are normally truncated with eclipses on hover [16:16:10] if you expand "group 1" and then scroll over to "noboard_chapter..." and hover it it will show the whole db name [16:17:12] mostly it was me goofing about with CSS :) [16:20:10] 10wikitech.wikimedia.org, 10Deployment-Systems, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3545398 (10mmodell) Using Phabricator calendar would work, and I'm all for it, if I can get consensus that it's... [16:21:31] 10wikitech.wikimedia.org, 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3545401 (10mmodell) p:05Triage>03Normal [16:23:34] 10wikitech.wikimedia.org, 10Deployment-Systems, 10Release-Engineering-Team (Backlog), 10User-MarcoAurelio: Create an easier way to add/remove/modify patches for SWAT - https://phabricator.wikimedia.org/T171940#3545411 (10mmodell) [16:30:02] 10Tools: Slow opening https://tools.wmflabs.org/commonshelper/ - https://phabricator.wikimedia.org/T173953#3545490 (10Zoranzoki21) [16:31:53] 10Tools: Slow opening https://tools.wmflabs.org/commonshelper/ - https://phabricator.wikimedia.org/T173953#3545516 (10Zoranzoki21) Other tools on tools.wmflabs.org are opening very fast. [16:32:28] 10Tools: Slow opening https://tools.wmflabs.org/commonshelper/ - https://phabricator.wikimedia.org/T173953#3545517 (10Zoranzoki21) I cleaned cash memory. [17:01:30] 10Striker: Warning while creating a new tool account - https://phabricator.wikimedia.org/T173958#3545676 (10bd808) [17:02:54] 10Striker: GID out of range warning while creating a new tool account - https://phabricator.wikimedia.org/T173958#3545692 (10bd808) [17:06:10] (03PS1) 10BryanDavis: Update service account GID range to match reality [labs/striker] - 10https://gerrit.wikimedia.org/r/373321 (https://phabricator.wikimedia.org/T173958) [17:08:08] (03CR) 10Andrew Bogott: [C: 032] Update service account GID range to match reality [labs/striker] - 10https://gerrit.wikimedia.org/r/373321 (https://phabricator.wikimedia.org/T173958) (owner: 10BryanDavis) [17:09:19] (03Merged) 10jenkins-bot: Update service account GID range to match reality [labs/striker] - 10https://gerrit.wikimedia.org/r/373321 (https://phabricator.wikimedia.org/T173958) (owner: 10BryanDavis) [17:12:57] 10Tools: Slow opening https://tools.wmflabs.org/commonshelper/ - https://phabricator.wikimedia.org/T173953#3545754 (10Zoranzoki21) 05Open>03Resolved I think it's solved, at least for now. If it slows down again, I'll reopen task. [17:32:18] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3546048 (10Cmjohnson) a:05Cmjohnson>03RobH @robh can you do the installs....let's get them accessible and then I will deal with the disk shelf is... [17:44:18] 10Striker, 10cloud-services-team (Kanban), 10User-bd808: Unable to add new LDAP user as maintainer of a tool - https://phabricator.wikimedia.org/T173963#3546209 (10bd808) [17:44:53] 10Striker, 10cloud-services-team (Kanban), 10User-bd808: Unable to add new LDAP user as maintainer of a tool - https://phabricator.wikimedia.org/T173963#3546260 (10bd808) New LDAP account: ``` $ ldap uid=strikertest2017-08-23 '*' '+' dn: uid=strikertest2017-08-23,ou=people,dc=wikimedia,dc=org objectClass:... [17:50:41] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3546297 (10bd808) [17:50:44] 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: Make potential for others to see IP Address for ssh sessions explicit in Toolforge membership request process - https://phabricator.wikimedia.org/T173845#3546295 (10bd808) 05Open>03Resolved a:03bd808 [17:51:11] 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10Technical-Debt, 10User-bd808: Replace deprecated phabricator conduit api calls in phabricator.py file - https://phabricator.wikimedia.org/T159044#3546311 (10bd808) 05Open>03Resolved [17:52:46] 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: Striker gives fatal error when a SUL account already in use tries to attach to a second LDAP account - https://phabricator.wikimedia.org/T164847#3546314 (10bd808) 05Open>03Resolved Now gives the error message "Wikimedia unifie... [17:54:15] 10Striker, 10cloud-services-team (Kanban), 10Patch-For-Review, 10User-bd808: Fatal error when adding a duplicate SSH key - https://phabricator.wikimedia.org/T167931#3546334 (10bd808) 05Open>03Resolved Now gives error message "Public key SHA256: already in use." [17:54:31] 10Tools: Slow opening https://tools.wmflabs.org/commonshelper/ - https://phabricator.wikimedia.org/T173953#3546339 (10Aklapper) 05Resolved>03declined [[ https://www.mediawiki.org/wiki/Bug_management/Bug_report_life_cycle| No code change, hence nothing resolved ]] :) [17:54:44] 10cloud-services-team (FY2017-18), 10Goal, 10Patch-For-Review, 10User-bd808: Perform initial Cloud Services rebranding - https://phabricator.wikimedia.org/T168480#3546342 (10bd808) [17:59:22] 10Striker, 10cloud-services-team (Kanban), 10Epic, 10Patch-For-Review, 10User-bd808: Manage shared tool accounts via Striker - https://phabricator.wikimedia.org/T149458#3546435 (10bd808) [17:59:24] 10MediaWiki-extensions-OpenStackManager, 10Striker, 10cloud-services-team (Kanban), 10User-bd808: Unable to add service group to service groups - https://phabricator.wikimedia.org/T128400#3546431 (10bd808) 05Open>03Resolved a:03bd808 This is fixed in #striker. As soon as a couple of bugs with maintai... [18:00:00] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3546437 (10madhuvishy) The interface flapping issue was because of a mis-connected cable, which @Cmjohnson's fixed now. Both management interfaces ar... [18:03:16] 10Tools: Slow opening https://tools.wmflabs.org/commonshelper/ - https://phabricator.wikimedia.org/T173953#3546478 (10Zoranzoki21) Ok [18:21:37] could someone delete job 7443163 from the grid? apparently it's been stuck for a month. I tried qdel but now it's just in status "dt" [18:22:24] stuck for a month d'oh [18:23:12] !help ^^ [18:23:12] legoktm: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [18:23:47] * TabbyCat wonders where all this "goat" stuff comes from [18:24:44] legoktm: i got it, doing in a sec [18:24:45] legoktm: have you tried to kill the job directly in the server in which the job is running? (tools-exec-...) [18:25:10] TabbyCat: I didn't. And https://phabricator.wikimedia.org/T173313 [18:25:12] madhuvishy: thanks [18:26:00] legoktm: done [18:26:08] ty! [18:26:17] madhuvishy: and, how did you do that? is it something I can do myself in the future? [18:26:43] legoktm: the admin group can do qdel -f [18:26:56] which is root forcing deletion of a job [18:27:02] ah, ok [18:27:21] * TabbyCat creates #catification :P [18:27:35] (i got myself added to that group 1 day before I joined the team so I could qdel -f :P) [18:29:14] heh [18:40:54] I'm quite new to this, would anyone mind helping me out with setting up a pywikibot to run on the cloud? [18:41:55] I'm looking to copy over some configuration files but can't figure out how [18:50:10] kolbert: you can use paws which just does that [18:50:14] paws.wmflabs.org [18:50:29] and login with meta-wiki oauth [18:50:33] When I try that I've gotten a redirect loop error [18:50:52] after pushing "start my server" [18:50:57] kolbert: I see, let me check [18:58:07] andrewbogott: I'm getting an error on horizon trying to do pretty much anything "An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator." [18:58:15] hello madhuvishy [18:58:17] for example loading https://horizon.wikimedia.org/project/instances/1b929704-b7d7-473f-8c55-ebbf24fb9dff/ [18:58:33] madhuvishy: new paws cluster is at tools-paws-master-01 [18:58:48] yuvipanda: hey! kolbert here is trying to get to paws [18:59:05] o/ [18:59:09] hi kolbert [18:59:54] XioNoX: I will have a look [19:00:08] thanks [19:01:06] kolbert: can you try again just now? [19:01:11] madhuvishy: I restarted hub pod [19:01:20] yuvipanda: oh cool okay [19:01:30] sure thing [19:01:44] ah I see [19:01:48] I see you're going into a crashloopbackoff [19:01:51] XioNoX: is there a long delay before you get that message? [19:02:01] andrewbogott: yes [19:02:08] aha found it [19:02:14] PermissionError: [Errno 13] Permission denied: '/home/paws/.jupyter' [19:02:41] kolbert: try again/ [19:02:45] ? [19:03:28] Ah! I seem to be in now [19:04:10] kolbert: awesome [19:04:18] 10Cloud-VPS, 10cloud-services-team (Kanban), 10WMF-Legal: Add warning about ssh ip address visibility to Cloud VPS TOU - https://phabricator.wikimedia.org/T173846#3546693 (10ZhouZ) [19:04:41] thank you [19:04:50] yuvipanda: did you have to fix anything for the perms? [19:13:15] XioNoX: I can reproduce but not sure what's happening yet. I'll let you know :) [19:13:30] thanks [19:16:55] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3546755 (10madhuvishy) Current status: We are not really sure why the disk shelves don't show up. As the next step, @Cmjohnson will try and call HP s... [19:19:13] madhuvishy: yeah, I ran a chown in screen :D [19:19:24] madhuvishy: I'll try to debug more properly later, my tutorial just finished [19:19:50] yuvipanda: okay :) thanks a ton! [19:20:03] madhuvishy: np! Thanks for pinging me! [19:20:07] hope your tutorial went well [19:20:26] madhuvishy: it was very well received! [19:20:36] yuvipanda: I'm not surprised :) [19:24:28] andrewbogott: I was ablet to launch an instance via horizon, not sure if it comes up tho yet :) [19:24:49] I think it's just the instance status pages that are broken… I have a fix ready [19:24:58] Which I can't test because of aforementioned puppet compiler issue https://puppet-compiler.wmflabs.org/compiler02/7582/californium.wikimedia.org/prod.californium.wikimedia.org.err [19:25:36] That failure only makes sense to me if the compiler does not regard californium.wikimedia.org as in eqiad... [19:26:20] chasemp: any idea about ^^ ? [19:26:48] that's definitely related to refactor, that would come from the fake labs private repo [19:26:57] right, but it's in there [19:26:59] I wonder if maybe that repo is not part of the puppet compiler setup? [19:27:05] it is yep [19:27:20] or wait this is californium ok, I was thinking instance [19:27:29] In theory the labs/private repo is the only reason the puppet compiler ever works for anything I think [19:28:09] * chasemp nods [19:29:21] andrewbogott: so I may have a thought sec [19:31:47] andrewbogott: the only thing I can thik of is that this is an interaction between the differing puppet fulfillment hiera trees, that value is in there but depending on how the compiler attempts to fulfill it it still may not be found. The value there now is found by instances on run using the labs master hiera tree, but it is not in the same ....path in hieradata dir that production hosts find it [19:32:05] becuase there is no overlap in the hiera tree between prod and cloud really while common is not expand_paths [19:32:30] chasemp: I started to c/p the setting into common.yaml but then I felt bad about changing-hiera-without-understanding-hiera [19:32:44] so I wonder if that's fixed by moving it to the same path that a prod host would find it in, whereas the path in labs/private now is how instances find it [19:33:16] wouldn't a prod host find it in eqiad.yaml? [19:33:17] yeah that's not a good solution as prod wouldn't find it there anyway unless it's using hiera_hash() [19:33:32] well, so it gets weird ok [19:33:42] if you use the role backend there are some magic things to it that chagne where it looks for things [19:33:52] but those same magic overlays are not done w/ hiera_hash() [19:33:58] so it depends on how you look [19:34:04] the tree is interpreted differently [19:34:12] it's pretty confusing [19:34:22] so [19:34:41] probably no, a normal role() lookup would not find that path in eqiad.yaml I don't think [19:34:58] as it does the path expansion and just like it never looks at common.yaml directly I don't think (but need to confirm) it wouldn't look in eqiad.yaml [19:35:11] because it would look in eqiad/profile/openstack/main.yaml instead [19:35:25] So it needs to be set under role/ someplace [19:35:51] yuvipanda, Hey there, I'm having a bit of trouble figuring out how to configure the bot [19:36:04] or 'profile'? [19:36:11] andrewbogott: no :) [19:36:15] sorry no was to the first [19:36:22] so that value for prod comes from [19:36:24] I've uploaded some text files for it to refer to (what replacements to make, what articles to edit) [19:36:40] but I don't think I have it set up properly [19:36:56] andrewbogott: hieradata/eqiad/profile/openstack/main.yaml [19:37:00] in in there you want [19:37:07] profile::openstack::main::observer_password: 'foo' [19:37:25] ok, will try [19:37:28] that's how prod w/ role wrapper but looking for that key works out [19:38:56] andrewbogott: potentially my fault for not taking the compiler into accounts, in so much as I was thinking of actual fulfillment in cloud and in prod and not the fact that an instance compiler would need shadowed prod values even if they are not valid in the labs hiera tree [19:39:18] (03PS1) 10Andrew Bogott: just trying to get the puppet compiler to find profile::openstack::main::observer_password [labs/private] - 10https://gerrit.wikimedia.org/r/373355 [19:39:24] chasemp: ^ ? [19:39:43] I may need to explain on hangout or something, it's deeply confusing [19:40:03] let's see if filling in that value works out [19:40:31] (03CR) 10Andrew Bogott: [V: 032 C: 032] just trying to get the puppet compiler to find profile::openstack::main::observer_password [labs/private] - 10https://gerrit.wikimedia.org/r/373355 (owner: 10Andrew Bogott) [19:41:00] kolbert: #pywikibot might be a better fit for that, let's take it there? [19:41:16] andrewbogott: hieradata/eqiad.yaml:profile::openstack::main::observer_password: 'Fs6Dq2RtG8KwmM2Z' is found via instances during lookup, that same value would not be found by a prod host via lookup even with same key path becuse of how the heira tree is interpreted [19:41:50] it worked [19:41:54] but the path of hieradata/eqiad/profile/openstack/main.yaml with key profile::openstack::main::observer_password: 'foo' even though it cannot be found by instances during actual puppet compilation is needed here (I think) for the compiler because it's shadowing production logic [19:41:56] I hate that these things are different [19:42:22] this really through me for a loop at the time, I understand it I think but yes [19:42:23] it's horrible [19:42:40] sorry I didn't think about the compiler and add those keys in to be complete [19:42:53] there are probably more missing just for completeness [19:43:14] yeah, I'll try a few other hosts after I get this thing fixed in horizon [19:43:56] andrewbogott: fwiw dding it to common.yaml would not have worked either [19:44:59] is this a weird case because it's buried::down::in::another::structure? [19:45:27] that's the effective reason yes [19:45:33] the reason for that is what is more nuanced [19:49:04] andrewbogott: this got clarified for me when I did early refactor and values in the hiera tree were overlooked, figurd out hiera_hash() behaviors differently from hiera() in interpreting where to look and that's why our old stuff works even within the role() function -- it's effectively ignoring it. But w/ the new paradigm of hiera() lookup as a class variable in a profile the path expansion foo::bar::down::what foo/ [19:49:04] bar/down.yaml with foo::bar::down::what: 'hey' is honored [19:49:13] and then the labs master tree is different still [19:49:15] :) [19:50:22] bd808 was nice enough to walk through it with me at like 8 pm one night because I thought I was losing my mind [19:50:41] it's a train wreck :/ [19:50:58] ok, I think I almost understand that. At least the difference between using hiera_hash and not [19:51:08] role() does weird things that are very unexpected [19:51:19] apparently by design [19:51:34] andrewbogott: example is hiera_hash() would find the value in common.yaml where hiera() would not [19:52:26] that makes sense, I think [19:52:42] Hm, I think my bug is still not fixed :( [19:54:54] 10cloud-services-team (Kanban), 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3546880 (10Cmjohnson) h/w log shows Record: 16 Date/Time: 08/15/2017 15:55:29 Source: system Severity: Critical Description: CPU 1 machine check error detecte... [20:21:54] 10Cloud-Services, 10Tools, 10Commons: Provide service to filter over categorization from a list of Commons categories - https://phabricator.wikimedia.org/T110833#1587559 (10Multichill) [20:55:10] 10Horizon, 10cloud-services-team (Kanban): puppet configuration broken on Horizon - https://phabricator.wikimedia.org/T173982#3547043 (10Andrew) [21:04:51] 10PAWS: 504 Gateway Time-Out when loading PAWSe - https://phabricator.wikimedia.org/T173983#3547077 (10kolbert) [21:05:21] 10PAWS: 504 Gateway Time-Out when loading PAWS - https://phabricator.wikimedia.org/T173983#3547077 (10kolbert) [21:44:31] andrewbogott: how goes it w/ teh horizon fix? [21:44:46] slowly but I'm not stuck yet [21:49:30] chasemp: could use a hand with this erb that doesn't compile: https://gerrit.wikimedia.org/r/#/c/373386/1/modules/role/templates/labs/puppetmaster/extra_auth_rules.conf.erb [21:50:11] andrewbogott: looking (meeting in 10 tho fyi) [21:50:43] probably just a quoting bug, updating the patch now [21:53:34] nope, still wrong [21:54:53] 21:51:00 modules/role/templates/labs/puppetmaster/extra_auth_rules.conf.erb:5: syntax error, unexpected ')' [21:54:54] 21:51:00 ...s.flatten(1).each do |worker| ).to_s); _erbout.concat "\nall... [21:56:24] then [21:56:26] 21:51:00 modules/role/templates/labs/puppetmaster/extra_auth_rules.conf.erb:8: syntax error, unexpected end-of-input, expecting ')' [21:56:46] my parens match up though don't they? [21:56:52] to my eye yes [21:57:05] this is a case of a bad error message I imagine [21:58:28] andrewbogott: I'm no tsure what to do except scale it back to just the lookup and then add components until it fails [21:59:13] ok, let's try without the loop... [21:59:34] (hopping on a call) [22:00:07] 10Striker, 10cloud-services-team (Kanban), 10User-bd808: Unable to add new LDAP user as maintainer of a tool - https://phabricator.wikimedia.org/T173963#3547272 (10bd808) I was able to add "Striker test 2017-08-23" as a maintainer via https://wikitech.wikimedia.org/wiki/Special:NovaServiceGroup so this looks... [22:03:01] yep, it's the for loop that breaks it [22:23:41] ok, I've hacked up a fix for that problem but something is still amiss. I need to go in a minute but I'll try to work on this more later tonight. [22:23:52] And also I hate erb [22:43:57] andrewbogott: is that still about the issue I reported? [22:53:08] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3471092 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by madhuvishy on neodymium.eqiad.wmnet for hosts: ```...