Azure Data Engineer

First Banks Inc

1. • Analyze, design, and build Modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes. • Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards. • Creating internal and external stage for data load. • I have created stored procedures, functions, views where I used them in my ETL operations in order to extract data from source. • Loading the data from flattened file to snowflake database and developing framework and database structures in snowflake. • Developing sql queries using SnowSql • Creating internal and external stage for data load. • Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster. • Hands-on experience with Snowflake utilities, SnowSQL, SnowPipe, Big Data model techniques using Python / Java. • Compared Self hosted Hadoop with respect to GCPs Data Proc, and explored Big Table (managed HBase) use cases, performance evolution. • ETL pipelines in and out of data warehouse using combination of Python and Snowflakes SnowSQL Writing SQL queries against Snowflake • Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in GCP. • Developed data warehouse model in snowflake for over 100 datasets using whereScape. • Heavily involved in testing Snowflake to understand best possible way to use the cloud resources. • Developed ETL workflows using NiFl to load data into Hive and Teradata. • To meet specific business requirements wrote UDF’s in Scala and Pyspark. • Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Sql Activity. • Worked on a project where we migrated exiting project that is from Hadoop to big query and then we are migrating our entire project to GCP • Hands-on experience on developing SQL Scripts for automation purpose. • Created Build and Release for multiple projects (modules) in production environment using Visual Studio Team Services (VSTS).

JNTUH

Computer Science