Design, build, and scale batch and real-time ETL/ELT pipelines and data platforms on AWS. Implement lakehouse architectures, optimize performance and cost, enforce governance and security, maintain observability, automate deployments with IaC/CI-CD, and collaborate with stakeholders to deliver scalable data solutions.
Deep expertise in AWS data, streaming, and compute services, including Kinesis, Flink, Elasticsearch, PostgreSQL, Lambda / Fargate, Step Functions, S3, Apache Iceberg, Glue, Datadog, and Redshift Spectrum Strong experience in real-time and batch data processing architectures, including event-driven and microservices-based data platforms Advanced proficiency in SQL, Python, and Spark for large-scale data processing, transformations, and analytics Hands-on experience with CI/CD pipelines, Git, and Infrastructure-as-Code (Terraform / CloudFormation) for automated deployments Proven expertise in workflow orchestration tools (Airflow, Dagster, Step Functions) and pipeline scheduling Strong understanding of data modeling techniques (dimensional, normalized, and lakehouse architectures) Deep knowledge of partitioning strategies, indexing, and query performance optimization Experience with data lake and lakehouse architectures (S3 + Iceberg / Delta-like patterns) Familiarity with observability and monitoring tools (Datadog, CloudWatch) for pipeline health and performance tracking Knowledge of data governance, lineage, cataloging, and metadata management frameworks Understanding of security and access control mechanisms (IAM roles, RBAC, encryption, data masking) Experience with cost optimization strategies in AWS (storage tiering, compute optimization, efficient query design) Roles and Responsibilities 1. Data Platform Engineering Design, build, and scale robust ETL/ELT pipelines for structured and unstructured data sources Implement both batch and real-time data ingestion frameworks Ensure pipelines are fault-tolerant, reusable, and scalable 2. Data Architecture & Design Define and implement end-to-end data architecture across ingestion, processing, storage, and consumption layers Drive adoption of lakehouse architecture and modern data patterns Establish data contracts and schema evolution strategies 3. Performance & Optimization Optimize pipelines and queries for high performance, cost efficiency, and scalability Analyze and resolve bottlenecks in processing, storage, and query execution Implement efficient partitioning, indexing, and caching strategies 4. Data Governance, Security & Compliance Implementation of data governance frameworks, including:
Data quality checks
Data lineage and traceability
Metadata management
Ensure compliance with security standards and regulatory requirements Implement fine-grained access control, encryption, and auditing mechanisms 5. Observability & Reliability Engineering Build and maintain monitoring, logging, and alerting frameworks Define and track SLAs / SLIs / SLOs for data pipelines Ensure high availability, fault tolerance, and incident response readiness 6. Automation & DevOps Practices Develop and maintain CI/CD pipelines for data applications Enable infrastructure automation using Terraform / CloudFormation Promote DevOps best practices across data engineering workflows 7. Stakeholder Collaboration & Solutioning Collaborate with business stakeholders to translate requirements into scalable technical solutions Partner with analytics, risk, and reporting teams to ensure data usability and accessibility Provide technical guidance and feasibility analysis for new use cases
Data quality checks
Data lineage and traceability
Metadata management
Ensure compliance with security standards and regulatory requirements Implement fine-grained access control, encryption, and auditing mechanisms 5. Observability & Reliability Engineering Build and maintain monitoring, logging, and alerting frameworks Define and track SLAs / SLIs / SLOs for data pipelines Ensure high availability, fault tolerance, and incident response readiness 6. Automation & DevOps Practices Develop and maintain CI/CD pipelines for data applications Enable infrastructure automation using Terraform / CloudFormation Promote DevOps best practices across data engineering workflows 7. Stakeholder Collaboration & Solutioning Collaborate with business stakeholders to translate requirements into scalable technical solutions Partner with analytics, risk, and reporting teams to ensure data usability and accessibility Provide technical guidance and feasibility analysis for new use cases
Similar Jobs
Fintech • Financial Services
Design, deploy, and maintain scalable, secure cloud infrastructure; automate provisioning and CI/CD pipelines; monitor and troubleshoot production systems; enforce security and compliance; support cloud migrations, disaster recovery, and platform upgrades; document runbooks and mentor team members in DevOps practices.
Top Skills:
AnsibleAWSAzureAzure DevopsAzure MonitorBashCloudwatchDockerElk StackGCPGithub ActionsGrafanaHelmIamJenkinsKubernetesPrometheusPythonTerraform
Cloud • Information Technology • Professional Services • Consulting
Design, develop, and deploy AWS cloud-native and serverless applications. Build Lambda-based APIs, microservices, and CI/CD pipelines; integrate with AWS services and third-party tools; implement monitoring, logging, and alerting; troubleshoot production issues; collaborate with customers and teams to meet technical requirements and improve architecture.
Top Skills:
Amazon S3Api GatewayAws LambdaAws SdksCi/Cd PipelinesDynamoDBMicroservicesServerless
Artificial Intelligence • HR Tech • Professional Services • Software
Design, build and optimize scalable AWS data ingestion, transformation and storage pipelines using Glue, Lambda, Kinesis, S3 and Redshift. Implement event-driven architectures, automate infrastructure with CDK/Terraform, enforce data governance and security (IAM, KMS, Lake Formation), and collaborate with data teams to tune performance and cost.
Top Skills:
Amazon AthenaAmazon DynamodbAmazon KinesisAmazon RedshiftAmazon S3Amazon SqsApache KafkaAws CdkAws CodepipelineAws GlueAws Glue DatabrewAws IamAws Lake FormationAws LambdaAws Step FunctionsEmrJavaKinesis Data StreamsKinesis FirehoseKmsPysparkPythonSQLTerraform
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
