Build and maintain scalable AWS data pipelines, warehouses, schemas, and ingestion systems for large structured and unstructured datasets. Develop ETL/ELT workflows using AWS services including Glue, Lambda, S3, Redshift, and CodePipeline. Create dimensional models, analytics tools, and data marts while ensuring data quality, privacy, compliance, and performance. Collaborate with cross-functional stakeholders to resolve data issues, improve infrastructure, and deliver actionable business insights.
The role will be responsible for setting up the data warehouses
necessary to handle large volumes of data, create meaningful analyses, and
deliver recommendations to leadership.
Core
Responsibilities
- Create and
maintain optimal data pipeline architecture ETL/ ELT into structured data
- Assemble
large, complex data sets that meet business requirements and create and
maintain multi-dimensional modelling like Star Schema and Snowflake
Schema, normalisation, de-normalization, joining of datasets.
- Expert level
experience in creating a scalable data warehouse including Fact tables,
Dimensional tables and ingest datasets into cloud based tools.
- Identify,
design, and implement internal process improvements including automating
manual processes, optimising data delivery and re-designing infrastructure
for greater scalability.
- Collaborate
with stakeholders to ensure seamless integration of data with internal
data marts, enhancing advanced reporting
- Setup and
maintain data ingestion, streaming, scheduling, and job monitoring
automation using AWS services. Setup Lambda, code pipeline (CI/CD), Glue,
S3, Redshift needs to be maintained for uninterrupted
automation.
- Build
analytics tools that utilize the data pipeline to provide actionable
insight into customer acquisition, operational efficiency, and other key
business performance metrics.
- Work with
stakeholders to assist with data-related technical issues and support
their data infrastructure needs.
- Utilize GitHub
for version control, code collaboration, and repository management.
Implement best practices for code reviews, branching strategies, and
continuous integration.
- Create data
tools for analytics and data scientist team members that assist them in
building and optimising our product into an innovative industry leader
- Ensure data
privacy and compliance with relevant regulations (e.g., GDPR) when
handling customer data.
- Maintain data
quality and consistency within the application, addressing data-related
issues as they arise.
Requirements
Required
- 4-8 years of
relevant experience
- Advanced
working SQL knowledge and experience working with relational databases,
query authoring (SQL) as well as working familiarity with a variety of
databases and Cloud Data warehouse like AWS Redshift
- Experience in
creating scalable, efficient schema designs to support diverse business
needs.
- Experience
with database normalization, schema evolution, and maintaining data
integrity
- Proactively
share best practices, contributing to team knowledge and improving schema
design transitions.
- Develop data
models, create dimensions and facts, and establish views and procedures to
enable automation programmability.
- Collaborate
effectively with cross-functional teams to gather requirements,
incorporate feedback, and align analytical work with business objectives
- Prior Data
Modelling, OLAP cube modelling
- Data
compression into PARQUET to improve processing and finetuning SQL
programming skills.
- Experience
building and optimizing “big data” data pipelines, architectures and data
sets.
- Experience
performing root cause analysis on internal and external data and processes
to answer specific business questions and identify opportunities for
improvement.
- Experience
with manipulating, processing and extracting value from large disconnected
unrelated datasets
- Strong
analytic skills related to working with structured and unstructured
datasets.
- Working
knowledge of message queuing, stream processing, and highly scalable “big
data” stores.
- Experience
supporting and working with cross-functional teams and Global IT.
- Familiarity of
working in an agile based working models.
Preferred
Qualifications/Expertise
- Experience
with relational SQL and NoSQL databases, especially AWS Redshift.
- Experience
with AWS cloud services Preferable: S3, EC2, Lambda, Glue, EMR, Code
pipeline highly preferred. Experience with similar services on another
platform would also be considered.
Education:
- Bachelor’s or
master’s degree on Technology and Computer Science background
Similar Jobs
Artificial Intelligence • Fintech • Software • Financial Services
Lead migration from Matillion ETL to AWS-native data platforms. Design and optimize scalable pipelines using AWS Glue and Apache Airflow, build reusable ETL/ELT components, and work with Snowflake for data warehousing and performance tuning. Implement data quality, monitoring, security, governance, and compliance practices. Collaborate with architects and stakeholders, support CI/CD and DevOps processes, conduct code reviews, mentor engineers, and improve processing performance, cost, and scalability.
Top Skills:
Apache AirflowAWSAws GlueAws LambdaAws Step FunctionsCi/CdDevOpsGitMatillion EtlPysparkPythonShell ScriptingSnowflakeSnowpipeSnowsqlSparkSQL
Artificial Intelligence • Big Data • Cloud • Machine Learning • Software
Designs and operates enterprise data governance infrastructure across Snowflake and AWS. Responsibilities include RBAC, authentication, masking, row-level security, compliance automation, Terraform provisioning, access workflows, monitoring, SOX evidence, data sharing, Iceberg governance, orchestration, cost optimization, and Snowflake AI/ML governance. The role also supports platform reliability, self-service data access, and scalable identity integration using Okta and SCIM.
Top Skills:
AlationAmazon AthenaAmazon EventbridgeApache AirflowApache IcebergSparkAWSAws LambdaCollibraGithub ActionsIamMwaaOktaPythonS3ScimSnowflakeSnowflake CortexSnowpark MlSQLTerraform
Artificial Intelligence • Software • Consulting • Generative AI
Design, build, and maintain scalable AWS-based data pipelines and ETL processes. Collaborate with data scientists, optimize storage and retrieval for performance and cost, enforce data governance and security, troubleshoot pipeline issues, and recommend new data engineering technologies.
Top Skills:
Apache AirflowAWSDockerEmrGlueJavaKubernetesLambdaNoSQLPythonRedshiftS3ScalaSQL
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
