Sarath B
Data Engineer
Irving, TX, USA
0
Followers0
FollowingWith eight years of extensive and diverse experience as a data engineer in delivering innovative solutions across various domains and industries. Expertise in data modeling, designing and maintaining ETL pipelines using cloud services like Azure, AWS, and big data technologies like Hadoop, Kafka, Spark to tackle complex business challenges. Well-versed in a wide array of cloud services, including Azure Data Factory, Databricks, and AWS Services like Athena, Glue, Lambda, and more. Proficient in various SQL databases like MySQL, SQL Server and NOSQL databases such as DynamoDB, Cosmos DB, and Oracle DB, and utilized data warehouses like Azure Synapse, AWS Redshift, and Snowflake, and adept at creating impactful dashboards using visualization tools like QuickSight, Tableau, and Power BI. Possesses a strong automation skill set, leveraging Airflow, Bash scripting, and Cron jobs to streamline workflows. Experienced in Spark applications, SQL, Pyspark, Agile methodologies, and influencing and building strong relationships with stakeholders and architecture teams. Also well-versed in Hadoop ecosystem components and CI/CD pipeline tools.
Careers
Senior Data Engineer
Eli Lilly
Full time contract12/2022 -
- Developed batch pipeline by ingesting data from, On-premises SQL Server
- and Oracle Database into Data Lake Storage Gen2 using Azure Data Factory.
- Used transformation scripts in Data Factory by invoking Azure Databricks
- notebooks to perform data processing, data cleansing, and profiling using
- Spark SQL and PySpark.
- To Store and manage huge volumes of transformed and clean data, utilized
- Azure Synapse analytics data warehouse.
- Developed reusable data pipelines in Azure Data Factory to extract and load
- data from Synapse analytics into Snowflake.
- Performed complex SQL queries in Snowflake to extract valuable insights from
- data stored and make data readily available for analysis.
- Integrated Apache Kafka with Azure Stream analytics to enable real-time
- data streaming and analytics.
- Stored aggregated data sets into Azure Data Lake Storage Gen2, allowing for
- near real-time insights and analysis.
- Staged the API and Kafka Data (JSON format) into Snowflake DB by
- Flattening same for functional services.
- Implemented data migration of multilevel state data from SQL server to
- owflake using Python and SnowSQL.
- Configured and optimized Snowpipe to efficiently handle high-volume data
- streams, ensuring timely and accurate data ingestion into Snowflake data
- warehouse.
- Scalable data warehouse architecture was implemented by utilizing Delta
- Lake to ensure data quality.
Skills
SQLPythonGithubAzureAWS LambdaDatabricksMySQLPysparkNoSQL, HadoopDatabase Management
Experience5-8 years
Hourly rate$65/hr
Open to
remotehybridonsite
Welcome to Outdefine
A free tokenized community dedicated to connecting global tech talent with remote job opportunities. Our platform is designed to help you connect, learn, and earn in the tech industry while providing the chance to collect DEF tokens. Join our vibrant community today and explore a world of possibilities for your tech career!
Join for free