[07:00:46] <Majavah>	 !log tools.majavah-bot removing multiple large log files
[07:00:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.majavah-bot/SAL
[07:04:59] <Majavah>	 !log tools.majavah-bot mark task 6 as approved and add cronjob for it
[07:05:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.majavah-bot/SAL
[07:22:04] <matanya>	 !log cvn Matanya set flags +AV on Mark7747 in #cvn-wp-hr
[07:22:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cvn/SAL
[09:39:34] <Bigem>	 Hey there! :)  Can anyone help me how to use the -exceptinside parameter as depicted here: https://www.mediawiki.org/wiki/Manual:Pywikibot/replace.py ? Just a practical example would be enough. 
[09:49:31] <arturo>	 Bigem: I suggest you send and email to cloud@l.w.o
[09:49:42] <arturo>	 you may find other pywikibot users there
[09:51:32] <Bigem>	 I've never used emails like these before (with lists, for the lack of the right term in English) and I didn't know how to interact correctly in them. :/  
[09:57:15] <arturo>	 I think you can succeed if you just send an email with the same content as the question you wrote here in the channel
[09:58:43] <Bigem>	 What's the email? Just cloud@l.w.o ?
[09:59:51] <arturo>	 Wikimedia Cloud Services general discussion and support <cloud@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/cloud
[10:00:06] <arturo>	 I suggest you also subscribe to https://lists.wikimedia.org/mailman/listinfo/cloud-announce cloud-announce@l.w.o
[10:01:51] <Bigem>	 Okay then. Thank you! :)  I'll try there.
[10:23:05] <arturo>	 !log cloudinfra added jmm (moritz) as user & projectadmin
[10:23:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[11:05:31] <Amir1>	 !log codesearch restart codesearch
[11:05:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[12:15:37] <arturo>	 !log admin icinga-downtime toolschecker for 2h (T265288)
[12:15:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:15:42] <stashbot>	 T265288: Enable L3 routing on cloudsw nodes - https://phabricator.wikimedia.org/T265288
[12:19:17] <arturo>	 !log admin `root@cloudcontrol1005:~# neutron subnet-create --gateway 185.15.56.241 --name cloud-instances-transport1-b-eqiad1 --ip-version 4 --disable-dhcp wan-transport-eqiad 185.15.56.240/29` (T265288)
[12:19:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:19:52] <arturo>	 !log admin subnet 185.1.5.56.240/29 has id 7c6bcc12-212f-44c2-9954-5c55002ee371 in neutron (T265288)
[12:19:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:41:01] <arturo>	 !log admin `root@cloudcontrol1005:~# neutron router-gateway-set  --fixed-ip subnet_id=7c6bcc12-212f-44c2-9954-5c55002ee371,ip_address=185.15.56.244 cloudinstances2b-gw wan-transport-eqiad` (T265288)
[12:41:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:41:05] <stashbot>	 T265288: Enable L3 routing on cloudsw nodes - https://phabricator.wikimedia.org/T265288
[12:41:28] <arturo>	 !log admin `root@cloudcontrol1005:~# neutron subnet-delete dcbb0f98-5e9d-4a93-8dfc-4e3ec3c44dcc` (T265288)
[12:41:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:42:12] <arturo>	 !log admin restarted neutron l3 agent in cloudnet1003 bc it still had the old default route (T265288)
[12:42:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:06:06] <arturo>	 revi: everything working now?
[13:06:17] <revi>	 yeah, it seems so
[13:08:37] <arturo>	 thanks
[18:59:27] <ottomata>	 hm, hiya, 4 hours ago razzi created deployment-kafka-jumbo-3
[18:59:29] <ottomata>	 https://horizon.wikimedia.org/project/instances/31237e57-7a10-4a9b-be2a-864c8ca3f7cb/?tab=instance_details__log
[18:59:37] <ottomata>	 neither of us can ssh in
[19:01:45] <ottomata>	 i'm not sure why, both of us are projectadmins and members
[19:01:47] <ottomata>	 of deployment-prep
[19:02:11] <ottomata>	 razzi is seeing his ssh key denied, and i see 
[19:02:11] <ottomata>	 channel 0: open failed: administratively prohibited: open failed
[19:02:11] <ottomata>	 stdio forwarding failed
[19:02:11] <ottomata>	 kex_exchange_identification: Connection closed by remote host
[19:02:24] <ottomata>	 we can log in just fine to other deployment-prep hosts
[19:05:28] <ottomata>	 ah I'm also seeing my key auth fail
[19:06:45] <ottomata>	 andrewbogott: is there something we might be missing about new hosts and ssh keys?
[19:07:16] <andrewbogott>	 I doubt it but I'll lool
[19:07:19] <andrewbogott>	 *look
[19:07:27] <andrewbogott>	 This is in deployment-prep?
[19:07:30] <ottomata>	 ya
[19:07:33] <andrewbogott>	 (horizon links aren't transferrable for the most part)
[19:07:34] <ottomata>	 deployment-kafka-jumbo-3
[19:08:00] <ottomata>	 i can log into e.g. deployment-kafka-jumbo-2 just fine
[19:09:58] <andrewbogott>	 ottomata: did you look at the syslog for puppet failures?  And/or did you confirm that puppet actually works on e.g. deployment-kafka-jumbo-2?
[19:11:32] <andrewbogott>	 -2 looks ok but on the new one I see
[19:11:34] <andrewbogott>	 https://www.irccloud.com/pastebin/z26JYOKn/
[19:12:27] <ottomata>	 andrewbogott:  how can I get syslog? i tried  the horizon log interface buut only saw recent prometheus related job (crons?)
[19:12:42] <ottomata>	 OH view full log
[19:12:45] <ottomata>	 ok.
[19:12:46] <ottomata>	 :p
[19:13:05] <andrewbogott>	 that's what I did :)
[19:13:09] <ottomata>	 got it, thank you
[19:13:11] <ottomata>	 i think i see now
[19:16:30] <ottomata>	 razzi: ah the problem is that since kafka-jumbo is configured using prefix puppet, it is already applying the kakfa broker role
[19:16:39] <ottomata>	 and, that references global hiera to figure out what cluster it belongs to
[19:16:40] <ottomata>	 so
[19:16:49] <ottomata>	 you need to edit Project Puppet in deployment-prep Horizon
[19:17:13] <ottomata>	 and add deployment-kafka-jumbo-3 to the list of brokers in kafka_clusters.jumbo-deployment-prep there
[19:17:21] <razzi>	 Ah i see
[19:17:34] <razzi>	 Confusing error :P
[19:17:55] <ottomata>	 well its unfortunenate that that kept puppet from adding ssh keys!
[19:18:07] <ottomata>	 ideally we'd have been able to log in and exclude that from the possible problems
[19:18:17] <ottomata>	 i didn't realize you  could get full logs from the Horizon web UI
[19:18:28] <ottomata>	 that puppet error makes sense if you look at that line in kafka/broker.pp
[19:18:38] <ottomata>	 $id = $brokers[$::fqdn]['id']
[19:18:46] <ottomata>	 and trace where $brokers is set in the including profile/role
[19:19:02] <ottomata>	 andrewbogott:  thank you <3
[19:19:29] <andrewbogott>	 sure thing
[22:18:16] <awight>	 !log deployment-prep metawiki: Promoting User:Jan Dittrich into centralnoticeadmin...
[22:18:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL