[13:32:11] !log admin enabled DRBD stats collection on labstore100[4-5] T208446 [13:32:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:32:14] T208446: prometheus - collect DRBD statistics on cloudstore/labstore servers - https://phabricator.wikimedia.org/T208446 [14:21:10] !log osmit migrating project to eqiad1-r [14:21:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Osmit/SAL [14:26:45] !log petscan migrating project to eqiad1-r [14:26:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Petscan/SAL [14:33:40] hi! I'm trying to apply profile::rsyslog::kafka_shipper to prefix deployment-mediawiki- in horizon, from the interface I see logging_kafka_brokers: ["deployment-logstash2.deployment-prep.eqiad.wmflabs:9093"] Remove Class [14:33:58] though puppet isn't happy about that [14:34:00] parameter 'logging_kafka_brokers' expects an Array value, got String on node deployment-mediawiki-07.deployment-prep.eqiad.wmflabs [14:34:18] any way to pass arrays that way from horizon or I should go some other way? [14:36:36] godog, I'd avoid the parameters stuff in classes and put it in hiera if possible [14:37:03] otherwise you get type madness like this, I think IIRC the class parameters are all treated as strings or something along those lines [14:38:16] seems like that's the case yeah, this parameter is read from hiera but I don't think I can put it in in the "hiera" section at the bottom of prefix puppet [14:38:25] I'll commit it in puppet.git I suppose [14:42:14] thanks Krenair [14:42:51] godog: that sounds like a bug; params set in on a class or in the 'hiera' section down below should get handled just the same [14:43:08] do you mind opening a ticket? Probably things aren't getting parsed correctly [14:43:11] I think the types work as you'd expect in hiera but the class parameters are all taken as strings [14:43:23] ah, that could be [14:43:34] !log rcm migrating project to eqiad1-r [14:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [14:43:39] still a bug imo [14:44:24] yep [14:44:44] thanks andrewbogott, yeah I'll open a task [14:45:41] the other thing I was surprised by is that if I want to set e.g. "profile::rsyslog::kafka_shipper::kafka_brokers" which is also a class parameter I can't do so in the bottom "hiera" section (confirming now) [14:46:32] godog: what do you mean by "can't"? [14:55:34] you mean one takes precedence over the other? or there's a missing hiera call? [14:57:19] andrewbogott: nevermind I can add the variable to hiera at the bottom and then it gets parsed and removed from there and shown next to the class name, so that works as expected I think [14:57:39] nevertheless I'm not seeing the class applied, anyways I'll do a few more tests and open tasks [14:57:43] thanks both! [14:57:54] ok, great. The UI is confusing since hiera is only stored in one place but you can enter it in two :/ [14:58:47] indeed, I was also confused by prefix puppet because the prefix tab is reset to the first tab once a change is committed [15:08:11] ok this is the thing I just mentioned https://phabricator.wikimedia.org/T210497 [15:08:39] no it isn't [15:08:48] https://phabricator.wikimedia.org/T210507 this is [15:42:43] and https://phabricator.wikimedia.org/T210513 re: array vs strings [16:50:47] !log social-tools migrating project to eqiad1-r [16:50:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Social-tools/SAL [17:01:24] legoktm: so far we're not getting any admin responses on https://phabricator.wikimedia.org/T204692, and the deadline is coming up fast. Can you suggest some other communication channel to use, or a different person to contact? [17:14:26] (that's re: the social-tools project) [17:59:01] andrewbogott: I'll respond on the ticket, sorry [17:59:09] thanks! [18:36:17] So I'm having issues getting into Labs stuff... getting connection closed by remote host on everything (-vv'd it, it is accepting my key), I can get into all my other key based servers, I seem to be a member of Shell users. Halp? [18:38:22] Ah! I don't think I'm a member of "bastion" anymore. [18:52:59] !help (^) I think I need adding to the "bastion" project on Cloud VPS as I can't seem to access any instances (which I should be able to access) [18:53:00] Lcawte: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [18:53:14] mmm [18:53:29] Lcawte: checking [18:54:08] do you know which bastion are you using? [18:54:19] !log quarry triggered OOM killer on quarry-worker-02 9 times T188564 [18:54:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [18:54:26] T188564: Quarry should refuse to save results that are way too large - https://phabricator.wikimedia.org/T188564 [18:55:43] Lcawte: are you a member of any Cloud VPS project other than Toolforge? [18:55:59] oh.. social-tools [18:56:01] hmm [18:56:48] Yeah. That. I notice most other people have "bastion" as one of their projects, but I don't. I'm also not on the bastion project page as a user. [18:57:10] yeah, something weird seems to have happened [18:57:36] its easy enough to add you manually back to the bastions project. [18:58:20] arturo: I think the fix is just making Lcawte a member of the bastion project (again). At least on the LDAP side that seems to have gone missing. [18:59:03] ok [18:59:38] checking [19:00:46] membership in that project is supposed to be granted automatically by a hook in keystone that fires when group membership changes. There is no reconciliation loop for that though, so if it gets out of sync for some reason it stays broken [19:01:03] I can't remember how we manage our LDAP [19:01:41] I'm sure I had access at some point, which is weirder (probably - I don't follow this stuff closely enough to know if you moved/revoked it from people for a reason) [19:02:53] arturo: I'll do it the sneaky way (with deprecated OpenStackManager pages) [19:04:30] errr... nope. that secret backdoor is apparently closed now [19:05:40] trying now in seaborgium [19:06:20] arturo: "openstack role add --user ..." should do it [19:06:27] ok [19:06:37] probably easier and safer than hacking ldap directly :) [19:06:39] that one I can do it without docs :-) [19:08:21] Lcawte: try now [19:10:43] I'm in! [19:10:46] Thank you :) [19:17:09] cool [19:55:52] !log project-proxy moving traffic to novaproxy-02 [19:55:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [19:57:56] !log project-proxy moving novaproxy-01 to labvirt1001 [19:57:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [20:14:23] !log moving traffic back to novaproxy-01 [20:14:24] andrewbogott: Unknown project "moving" [20:14:41] !log project-proxy moving traffic back to novaproxy-01 [20:14:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL