Data Engineer - Cloud & Big Data

Role Summary:

Responsible for designing, developing, and maintaining scalable cloud-based data pipelines and data platforms across AWS, GCP, Kafka, and Databricks. The role involves building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization.

In this role you will...

• Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark.
• Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka.
• Develop CDC-based data ingestion and event-driven data processing.
• Work with AWS services including Glue, S3, Lambda, IAM, KMS, Secrets Manager, and related services.
• Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, Cloud Functions, and related services.
• Implement data processing using Databricks, PySpark, Delta Lake, Bronze/Silver/Gold architecture, and DLT.
• Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance.
• Implement data quality, validation, monitoring, logging, and alerting across data pipelines.
• Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure.
• Optimize pipeline performance and cloud infrastructure costs.
• Implement secure data solutions using IAM, encryption, network controls, secrets management, and least-privilege access.
• Collaborate with application, DevOps, QA, analytics, and product teams to deliver reliable data solutions.

You'vegot what it takes if you have...

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with AWS + Databricks + Kafka.
• Experience working in multi-cloud environments (AWS/GCP).
• Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance.
• Experience troubleshooting high-volume production data pipelines.
• Experience with cloud performance and cost optimization.

#LI-Onsite