Data Migration and Building CLoud Native Data Lake

Industry: Insurance

Problem Context

Client is a large insurance company and their data is stored in multiple formats which resulted in data duplicacy. The client sought to build a centralised cost-effective and scalable data platform hosting for low-latency delivery to enterprise customers around the world with self-service BI capabilities.

Challenges

 Use of Different Data ingestion and ETL techniques for streaming and historical data.

 Due to GDPR and compliance issues data lake had to be built in Ireland region on AWS.

 Business had around 1600 tables and a lot of attributes to work with for reporting.

 Absence of proper Data dictionary and Data Lineage tracking feature.

Technologies Used

Amazon EC2

AWS Lambda

Amazon Redshift Spectrum

Amazon DMS

Amazon Kinesis

Amazon SQS

Amazon SNS

Apache Kafka

Apache S3

Power BI

Solution

S&M Tech Solutions helped them migrate data from multiple on-site data sources to the cloud and established a cost-effective, scalable and centralized data lake platform on the cloud by setting up an intake pipeline with security protocols and enabling them to view and gain insights in real time.

Result

 Successfully built a cost effective Data Lake for a quicker delivery to customers around the world

 The solution was able to reduce the time to market by 70%