Lead migration from Matillion ETL to AWS-native data platforms. Design and optimize scalable pipelines using AWS Glue and Apache Airflow, build reusable ETL/ELT components, and work with Snowflake for data warehousing and performance tuning. Implement data quality, monitoring, security, governance, and compliance practices. Collaborate with architects and stakeholders, support CI/CD and DevOps processes, conduct code reviews, mentor engineers, and improve processing performance, cost, and scalability.
We are seeking an experienced Senior
AWS Data Engineer to lead and execute a large-scale migration from Matillion
ETL to a modern AWS-native data platform. The ideal candidate will possess
deep expertise in Apache Airflow, AWS Glue, AWS Lambda and Snowflake,
with a strong background in designing scalable data pipelines, cloud-based data
integration, workflow orchestration, and data warehouse modernization.
The candidate will work closely
with business stakeholders, data architects, and engineering teams to migrate
existing ETL workloads, optimize data processing frameworks, and establish best
practices for cloud-native data engineering.
Requirements
Key Responsibilities
- Lead
the migration of existing Matillion ETL pipelines to AWS-native solutions.
- Participate
in Requirements Gathering and Analysis of existing data processing jobs.
- Design,
develop, and promote scalable data pipelines using AWS Glue and Apache
Airflow (AWS Service).
- Build
robust data ingestion, transformation, and orchestration frameworks on
AWS.
- Create
reusable and modular ETL/ELT components for enterprise data platforms.
- Data
modeling and warehousing concepts – must have
- Data
migration and modernization projects – good to have
- Implement
data quality, monitoring, alerting, security, governance, and compliance
standards.
- Collaborate
with data architects and business teams to understand migration
requirements and translate them into technical solutions.
- Optimize
data processing performance, cost, and scalability across AWS services.
- Participate
in CI/CD best practices and DevOps processes for data engineering
workflows.
- Perform
code reviews, mentor junior engineers, and drive engineering excellence.
Mandatory Skills
Core Technologies
- Apache
Airflow (DAG development, workflow orchestration, scheduling,
monitoring)
- AWS
Glue (ETL jobs, Glue Studio, Crawlers, Data Catalog, Spark-based
transformations)
- Snowflake (Data Warehousing, SnowSQL, Snowpipe, Streams & Tasks, Performance
Tuning)
- AWS Lambda
and Step Functions (preferred)
- Git
experience
Programming languages: Python,
SQL, Spark (must)
PySpark, Matillion, Shell
scripting (good to have)
Experience Requirements
- 8+
years of overall data engineering experience.
- 4+
years of hands-on experience with AWS data services.
- Proven
experience migrating ETL platforms such as Matillion, Informatica, Talend,
or DataStage to AWS-native architectures (Good to have).
- Strong
experience developing workflows in Apache Airflow.
- Extensive
experience building and optimizing AWS Glue jobs.
- Hands-on
experience with Snowflake administration and performance tuning.
- Strong
understanding of cloud architecture, scalability, deployments.
Similar Jobs
Healthtech • Pharmaceutical • Manufacturing
Design, build, and maintain scalable AWS-based data platform services and high-performance data pipelines. Provide technical leadership, establish data engineering best practices, and partner with cross-functional teams to deliver data integration, transformation, governance, and physical data models for healthcare solutions.
Top Skills:
Aurora PostgresqlAws EmrAws LambdaData LakehouseData MeshDynamoDBGluePysparkPythonRdsRedshiftS3ScalaSQLTerraform
Artificial Intelligence • Software • Consulting • Generative AI
Design, build, and maintain scalable AWS-based data pipelines and ETL processes. Collaborate with data scientists, optimize storage and retrieval for performance and cost, enforce data governance and security, troubleshoot pipeline issues, and recommend new data engineering technologies.
Top Skills:
Apache AirflowAWSDockerEmrGlueJavaKubernetesLambdaNoSQLPythonRedshiftS3ScalaSQL
Financial Services
Leads multiple software engineering teams, providing technical direction, coaching, and oversight. Owns resource and budget decisions, SDLC practices, cross-team collaboration, operational excellence, and delivery execution. Drives enterprise adoption of AI-assisted engineering and SDLC automation with governance, security, resiliency, quality, and measurable outcomes. Advises leadership on technical strategy and efficiency while hiring, developing, and recognizing talent.
Top Skills:
AngularCloud-Native TechnologiesDjangoFlaskPythonReact
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
