Gayathri G

Data Architect

Hi Team,
I'm Gayathri G, I have 9+ years of IT experience and 7+ years of Big Data experience.    I've been working in big data ecosystems like Spark,Athena,Python,pyspark,Redshift, AWS kinases, and DynamoDB also with Azure Data Factory, Databricks, Azure Active Directory, Blob Storage, Data Lake, SQL Database, SQL Data Warehouse and Hadoop ecosystem such as Hive, Yarn, Spark, MapReduce, HBase, Spark, Zookeeper, Hadoop, Airflow, Stream Sets, Oozie, Sqoop, Flume, Pig, Impala, Kafka.  I also have certification in Microsoft Certified Azure Data Engineer Associate DP-203.  I have extensive experience with the programming languages Java, Scala, Impala, SQL, and Python. I have experience working with Batch Processing & Real-time streaming using KAFKA. I do have lead experience actually, in fact, my team size is basically six engineers. We have data engineers and also business teams as part of the team. 
I did my masters in Information systems from IUPUI. And completed my bachelor's degree in Computer science. 
In my current project, I’m working in Lambda architecture where we ingest data both in batch and real-time. For batch, we ingest data from Teradata and SQL Server land data in s3, write py-spark ETL scripts to read data from s3, and do all the computations in Spark and EMR, write data back into Hive and Athena, and I also export the latest snapshot data into Redshift for analytical purposes (now we are using snowflake), where we have visualization developers that build Tableau dashboards on top of that, for real-time I write Spark streaming applications where we ingest data from AWS kinases and write data into DynamoDB for API consumption. our downstream consumers are stakeholders, data scientists, and other engineering teams as well. We use airflow DAGs to schedule all the workflows we develop in Python, and that's pretty much about my experience in my current project as well.