Lead migration of Matillion ETL workloads to AWS-native data platforms. Design, develop, and optimize scalable pipelines using AWS Glue and Apache Airflow, build reusable ETL/ELT components, and work with Snowflake data warehouses. Responsibilities include data modeling, ingestion, transformation, orchestration, monitoring, data quality, security, governance, cost optimization, and CI/CD. Collaborate with architects, stakeholders, and engineering teams to translate migration requirements into reliable cloud data solutions.
We are seeking an experienced AWS Data Engineer to lead and execute a large-scale migration from Matillion
ETL to a modern AWS-native data platform. The ideal candidate will possess
deep expertise in Apache Airflow, AWS Glue, AWS Lambda and Snowflake,
with a strong background in designing scalable data pipelines, cloud-based data
integration, workflow orchestration, and data warehouse modernization.
The candidate will work closely
with business stakeholders, data architects, and engineering teams to migrate
existing ETL workloads, optimize data processing frameworks, and establish best
practices for cloud-native data engineering.
Requirements
Key Responsibilities
- Develop modules/code for migration of existing Matillion jobs to AWS-native solutions.
- Participate
in Requirements Analysis of existing data processing jobs.
- Design,
develop, and promote scalable data pipelines using AWS Glue and Apache
Airflow (and other basic AWS Services) - must have.
- Build
robust data ingestion, transformation, and orchestration frameworks on
AWS.
- Create
reusable and modular ETL/ELT components for enterprise data platforms.
- Data
modeling and warehousing concepts – must have
- Data
migration and modernization projects – good to have
- Implement
data quality, monitoring, alerting, security, governance, and compliance
standards.
- Collaborate
with data architects and business teams to understand migration
requirements and translate them into technical solutions.
- Optimize
data processing performance, cost, and scalability across AWS services.
- Participate
in CI/CD best practices and DevOps processes for data engineering
workflows.
Mandatory Skills
Core Technologies
- Apache
Airflow (DAG development, workflow orchestration, scheduling,
monitoring)
- AWS
Glue (ETL jobs, Glue Studio, Crawlers, Data Catalog, Spark-based
transformations)
- Snowflake (Data Warehousing, SnowSQL, Snowpipe, Streams & Tasks, Performance
Tuning)
- AWS Lambda
and Step Functions (preferred)
- Git
experience (must-have)
Programming languages: Python,
SQL, Spark (any 1 is must)
PySpark, Matillion, Shell
scripting (good to have)
Experience Requirements
- 2+
years of intensive hands-on experience with AWS data services.
- Proven
experience migrating ETL platforms such as Matillion (or other ETL tools like DataStage) to AWS-native architectures (Good to have).
- Strong
experience developing workflows in Apache Airflow.
- Extensive
experience building and optimizing AWS Glue jobs.
- Hands-on
experience with Snowflake stored procedures (must-have), performance tuning (good-to-have).
- Strong
understanding of cloud architecture, scalability, deployments.
Similar Jobs
Financial Services
Designs, develops, and maintains scalable batch and streaming data pipelines using Python, PySpark, Spark, Databricks, SQL, and AWS. Leads data modeling and solution design, builds high-volume ingestion and transformation workflows, optimizes distributed workloads, and applies data warehousing best practices. Performs debugging, query tuning, code reviews, version control, and CI-friendly development while partnering with stakeholders and validating AI-assisted engineering outputs.
Top Skills:
Amazon S3SparkAWSAws CloudformationBitbucketDatabricksGitInfrastructure As Code (Iac)PysparkPythonSQLTerraform
Fintech • Financial Services
Design, develop, and optimize scalable cloud data pipelines and Snowflake warehouses using AWS, Airflow, dbt, and SQL. Automate data workflows, infrastructure provisioning, deployments, migrations, and operational support. Monitor pipeline health, troubleshoot data quality and performance issues, and maintain secure, reliable data architectures. Collaborate with data scientists, analysts, and enterprise stakeholders on ingestion, transformation, integration, and analytics initiatives while documenting systems and procedures.
Top Skills:
Amazon Ec2Amazon S3Apache AirflowApache KafkaSparkAWSAws GlueAws LambdaAzureAzure DevopsCi/CdCloudFormationCloudwatchDbtGCPGitGithub ActionsGitlab CiGrafanaIamJenkinsKubernetesMySQLPostgresPrestoPrometheusPythonShell ScriptingSnowflakeSplunkSQLTerraform
Artificial Intelligence • Analytics • Consulting • Pharmaceutical
Build and maintain scalable AWS data pipelines, warehouses, schemas, and ingestion systems for large structured and unstructured datasets. Develop ETL/ELT workflows using AWS services including Glue, Lambda, S3, Redshift, and CodePipeline. Create dimensional models, analytics tools, and data marts while ensuring data quality, privacy, compliance, and performance. Collaborate with cross-functional stakeholders to resolve data issues, improve infrastructure, and deliver actionable business insights.
Top Skills:
Amazon Ec2Amazon EmrAmazon S3AWSAws CodepipelineAws GlueAws LambdaAws RedshiftEltETLGitNoSQLOlapParquetSQL
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
