Projects from AWS

KDA Studio Create widget

Introduction:

Amazon Kinesis Data Analytics is the easiest way to transform and analyze streaming data in real time using Apache Flink. It lets customers to write SQL on streaming data using interactive notebooks. Currently we provide connectors to process streaming data from KDS Streams and Amazon MSK.

The idea is to create a re-usable widget component to make it easy to integrate with Streaming services consoles

My Role:

Understanding the user needs, help creating the mockups for this project, meeting cross functional teams to understand their limitations and working closer with engineers before and after designs being handed over to developing.

User Stories

As a Streaming service user

Should be able to create Studio notebook.
Should be able to see results of sample query.
Should be able to view a notebook that was created in my previous session if I left it running

Design workshop

Started with the workshop including Product manager ,Engineers and Customer facing engineers to discuss about the customers needs and use cases we have

Designing mockups

Worked on mockups after collecting all the insights from the workshops and we also do meetings product manager and engineers twice a week to talk about the project, get it reviewed discuss the next steps

Amazon MSK Private link

Background:

Multi-VPC and cross-account connectivity is a vital aspect of several customer architectures where they use Amazon MSK as a central event bus, with data flowing in and out from client(s) that are owned by different teams and hosted in different Amazon VPCs/accounts.

Previously customers used networking solutions like
1. VPC Peering
2. Amazon Transit Gateway
3. Self managed Private link to achieve multi-VPC connectivity.

Painpoints:

1) Found it very hard to manage complex routing tables
2)Hit the scaling limits as these solutions require the connecting VPCs to non-overlapping Private IP address ranges.
3)Several configuration updates including DNS were required to maintain private connectivity every time new brokers were added to the cluster
4) Customers have asked for a managed solution that simplifies the networking requirements for achieving multi-VPC and cross-account connectivity.

Solution: Multi-VPC Private Connectivity

Our solution is to provide a multi-VPC private connectivity (powered by AWS PrivateLink) for Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters, a new connectivity feature that enables customers to easily connect their Kafka clients hosted in different Virtual Private Clouds (VPCs) and accounts to a central Amazon MSK cluster. With this feature, clients can simply use the newly generated PrivateLink specific broker bootstrap strings to privately connect to the central Amazon MSK cluster while keeping all traffic within the AWS Network

Customer benefits:

Allow customers to have a MSK managed solution that enables Kafka clients spread across one or more VPCs and AWS accounts to connect privately to your Amazon MSK cluster.
Eliminates the operational overhead of self-managing a PrivateLink solution and scales seamlessly as the Amazon MSK cluster scales, enabling you to maintain private connectivity to the cluster without making additional configuration changes.
For customers using solutions like VPC Peering and AWS Transit Gateway for achieving multi-VPC connectivity and are concerned about the scalability of these solutions given their underlying requirement to have non-overlapping IP addresses across connecting VPCs.

My Role:

Understanding the user needs, help creating the mockups for this project, meeting cross functional teams to understand their limitations and working closer with engineers before and after designs being handed over to developing.