[10:17:26] <godog>	 hi, FYI the ceph alerts in icinga are UNKNOWN; prometheus-labmon.eqiad.wmnet still points to cloudmetrics1002
[10:23:45] <arturo>	 mmmm
[10:23:58] <arturo>	 ok, let me fix that!
[10:26:12] <arturo>	 that name is unfortunate
[10:26:32] <arturo>	 I was only looking for cloud* stuff when failovering cloudmetrics1002 the other day
[10:28:02] <godog>	 yeah I noticed it is a mixture of names :|
[10:28:09] <godog>	 anyways thanks for investigating arturo 
[10:28:20] <Majavah>	 grafana-labs now asks me to login, but does not take my ldap creds
[10:28:36] <arturo>	 it should allow anon logins
[10:28:39] <arturo>	 however: `grafana-server[902]: t=2021-05-10T10:28:19+0000 lvl=eror msg="Anonymous access organization error: 'WMCS': organization not found"`
[10:28:56] <arturo>	 not sure how to fix that yet
[10:29:16] <arturo>	 this is merged now: https://gerrit.wikimedia.org/r/c/operations/dns/+/688213
[10:29:51] <Majavah>	 server admin -> orgs -> select current -> rename at least on my home grafana installation
[10:31:01] <arturo>	 I don't have any 'server admin' option
[10:31:44] <Majavah>	 :/
[10:32:03] <arturo>	 OK I think I fixed that one
[10:32:06] <arturo>	 then I have
[10:32:09] <arturo>	 `grafana-server[902]: t=2021-05-10T10:31:43+0000 lvl=eror msg="Could not find plugin definition for data source: datasource-plugin-genericdatasource"`
[10:32:21] <arturo>	 but all this makes me wonder, is this config not in puppet? :-S
[10:32:35] <arturo>	 if a database, isn't that replicated from the other server?
[10:33:47] <Majavah>	 https://grafana-labs.wikimedia.org/d/000000012/tools-basic-alerts?orgId=1&refresh=5m works, which is enough for me :P
[10:34:59] <Majavah>	 why are there so many erroring queues, I cleared those yesterday
[10:38:01] <arturo>	 godog: I see this:
[10:38:26] <arturo>	 https://usercontent.irccloud-cdn.com/file/fFGoYnpr/image.png
[10:38:44] <arturo>	 any idea how to update that tools.wmflabs.org URL
[10:38:45] <arturo>	 ?
[10:40:35] <Majavah>	 actually something was wrong with the replication, that tools basic alerts board has a graph about k8s pod restarts but it's missing
[10:41:03] <arturo>	 ok
[10:41:27] <arturo>	 with replication you mean... between cloudmetrics servers?
[10:42:15] <Majavah>	 I guess so? that dashboard is different on 1001 than what it was on 1002
[10:42:54] <arturo>	 I'm afraid all changes we made to grafana as it was on 1002 are now lost. I doubt we had any kind of grafana-spècific replication between 1002 <-> 1002
[10:42:59] <arturo>	 1001 <-> 1002
[10:43:42] <Majavah>	 not a big deal, those can probably be recreated
[10:43:43] <arturo>	 honestly this whole cloudmetrics observability stack needs love and attention
[10:43:57] * Majavah wonders what would be needed to be able to edit grafana dashboards
[10:44:06] * arturo brb
[10:44:44] <Majavah>	 https://github.com/wikimedia/puppet/blob/76e1f7cffcce02d4c303800eaa21a98698fdcd74/hieradata/role/common/wmcs/monitoring.yaml#L49
[11:00:17] <CFisch_WMDE>	 o/
[11:00:51] <CFisch_WMDE>	 ( wrong channel ^^' )
[11:02:23] <arturo>	 Majavah: aren't you in that group'
[11:02:24] <arturo>	 ?
[11:02:57] <Majavah>	 arturo: no
[11:04:04] <Majavah>	 I've signed some sort of nda, but since toolforge root does not directly require anything in it, I'm not in it automatically
[11:04:54] <Majavah>	 s/it/cn=nda/
[11:06:48] <arturo>	 ok
[11:50:15] <Majavah>	 !log toolsbeta testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta T264221
[11:50:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[11:50:20] <stashbot>	 T264221: Upgrade the nginx ingress controller in Toolforge (and likely PAWS) - https://phabricator.wikimedia.org/T264221
[12:03:36] <godog>	 arturo: mmhh if the datasource isn't editable from the UI then must be in /etc/grafana 
[12:53:28] <Majavah>	 !log tools creating tools-k8s-haproxy-[3-4] to rebuild current ones without nfs and with keepalived
[12:53:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:04:39] <CFisch_WMDE>	 !log deployment-prep Forward renamed config name for improved template search features (T277028)
[14:04:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[14:04:44] <stashbot>	 T277028: Add description to search and change title of component - https://phabricator.wikimedia.org/T277028
[14:04:53] <CFisch_WMDE>	 !log deployment-prep Improve comment around ReferencePreviews beta cluster default (T271206)
[14:04:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[14:04:56] <stashbot>	 T271206: Enable RefPreviews on first wikis - https://phabricator.wikimedia.org/T271206
[14:14:39] <andrewbogott>	 !log traffic fixing merge conflict in /labs/private 
[14:14:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Traffic/SAL
[14:15:05] <andrewbogott>	 !log traffic fixing some confusion about puppetmaster name (.wmflabs vs. .cloud)
[14:15:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Traffic/SAL
[14:30:11] <andrewbogott>	 tarrow: Docker has filled up the drive on orig-01.wikibase-registry.eqiad1.wikimedia.cloud and it's failing in various sad ways.  Can you do a cleanup there please?  (or delete the VM if it's no longer good for anything)
[14:30:22] <tarrow>	 right
[14:30:52] <tarrow>	 andrewbogott: I feel I had this task to do a few months ago and I failed to finish it in some way :(
[14:30:55] <tarrow>	 sorry :/
[14:31:17] <andrewbogott>	 It's not a problem, it just came up in a routine check of puppet breakage :)
[14:34:15] <tarrow>	 andrewbogott: I hit the nuke button
[14:34:18] <tarrow>	 thanks :)
[14:34:21] <andrewbogott>	 that works!
[14:35:53] <tarrow>	 yeah, the people who wanted it claimed that they don't need it any more. I'm happy to have less burden on us
[14:57:38] <arturo>	 !log tools allow tools-k8s-haproxy-[3-4] to use the tools-k8s-haproxy-keepalived-vip address (172.16.6.113) (T252239)
[14:57:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:57:42] <stashbot>	 T252239: Rebuild tools-k8s-haproxy-* as an anti-affinity server group - https://phabricator.wikimedia.org/T252239
[15:03:33] <Majavah>	 !log tools clear all error states caused by overloaded exec nodes
[15:03:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:06:35] <Majavah>	 !log tools carefully rolling out keepalived to tools-k8s-haproxy-[3-4] while making sure [1-2] do not have changes
[15:06:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:22:43] <Majavah>	 !log tools change k8s.svc.tools.eqiad1.wikimedia.cloud. to point to the tools-k8s-haproxy-keepalived-vip address 172.16.6.113 (T252239)
[15:22:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:22:47] <stashbot>	 T252239: Rebuild tools-k8s-haproxy-* as an anti-affinity server group - https://phabricator.wikimedia.org/T252239
[15:29:27] <Majavah>	 !log tools.fourohfour restarting, unresponsive
[15:29:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fourohfour/SAL
[16:05:50] <mutante>	 Was there any change in the last couple days that could explain why email sent from cloud VPS would not arrive ..but now it does?
[16:06:01] <mutante>	 anything with mail servers lately?
[16:06:19] <mutante>	 It's a good thing, it works now. just dont know why
[16:07:00] <Majavah>	 mutante: we replaced the cloud vps outbound mail servers a week or two ago, but I don't think we did anything in the last day or two that would explain it starting again
[16:08:09] <mutante>	 Majavah: sometime between April 30 and today made it work. so I guess that was it. Though normally wouldn't expect changes in mail routing behaviour
[16:08:39] <mutante>	 thanks, I will just accept it is resolved 
[16:38:11] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 248527544d (l10n updates)
[16:38:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[17:43:33] <Majavah>	 !log toolsbeta testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta T264221
[17:43:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[17:43:36] <stashbot>	 T264221: Upgrade the nginx ingress controller in Toolforge (and likely PAWS) - https://phabricator.wikimedia.org/T264221
[17:59:28] <wm-bot>	 !log tools.ranker <lucaswerkmeister> deployed 3c78f9f4b5 (remove dead code)
[17:59:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL
[19:31:23] <ragesoss>	 I'm going to do a switchover of the backend instance for outreachdashboard.wmflabs.org. Is that something I can do just by updating the existing web proxy from Horizon? When I click 'update proxy' for it, both the 'Record' and 'Domain' fields in the mobile and non-editable, but the Domain field incorrectly shows "wmcloud.org" rather than "wmflabs.org".
[19:33:27] <andrewbogott>	 ragesoss: looking...
[19:33:52] <andrewbogott>	 ragesoss: what project is that in?
[19:34:00] <ragesoss>	 ("in the modal are non-editable" I meant)
[19:34:05] <ragesoss>	 globaleducation
[19:35:08] <ragesoss>	 I can still set up a new proxy for wmflabs, and maybe showing the default wmcloud domain instead of the actual domain of the edited proxy is just a UI bug?
[19:36:02] <andrewbogott>	 yeah, it's definitely a UI bug I'm not sure if it will therefore create a new record or update the proper one
[19:36:13] <andrewbogott>	 I will look at the cause of the bug... in the meantime you can try it or not as you prefer :)
[19:36:31] <ragesoss>	 I guess I can test it out with a not-in-use proxy that's already there.
[19:37:11] <ragesoss>	 oh, it errors: Error: Unable to update proxy: No such domain Details
[19:37:11] <ragesoss>	 can only concatenate str (not "int") to str
[19:37:22] <andrewbogott>	 T282489
[19:37:23] <stashbot>	 T282489: Horizon proxy dashboard: edit dialog shows wmcloud.org even if a wmflabs.org proxy is being edited - https://phabricator.wikimedia.org/T282489
[19:37:37] <andrewbogott>	 ragesoss: that's all terrible, I will see if I can reproduce in a test env
[19:42:22] <bstorm>	 !log toolsbeta setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes
[19:42:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:54:20] <andrewbogott>	 ragesoss: try now?
[19:56:08] <ragesoss>	 andrewbogott: yep, that works now!
[19:56:26] <andrewbogott>	 great, sorry for the bug
[19:56:44] <ragesoss>	 no prob, thanks for the fix!
[22:58:28] <bstorm>	 !log tools setting `profile::wmcs::kubeadm::docker_vol: false` on ingress nodes
[22:58:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:58:49] <bstorm>	 !log tools cleared error state on a grid queue
[22:58:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:08:58] <wm-bot>	 !log tools.jouncebot <bd808> Restart to revert 560a22a (T243394)
[23:09:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL