The Senior Data QA Engineer validates ETL pipelines, data transformations, business logic, KPIs, metrics, and analytical datasets across Snowflake, Databricks, Hive, and other platforms. Responsibilities include designing test cases, developing automated data tests with SQL and Python, integrating testing into CI/CD workflows, performing regression and release validation, investigating production issues, conducting root-cause analysis, and improving data-quality frameworks and processes.
𝗧𝗵𝗶𝘀 𝗿𝗼𝗹𝗲 𝗶𝘀 𝗳𝗼𝗿 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗪𝗲𝗲𝗸𝗱𝗮𝘆'𝘀 𝗰𝗹𝗶𝗲𝗻𝘁𝘀
𝗦𝗮𝗹𝗮𝗿𝘆 𝗿𝗮𝗻𝗴𝗲: 𝗥𝘀 𝟭𝟬𝟬𝟬𝟬𝟬𝟬 - 𝗥𝘀 𝟭𝟲𝟬𝟬𝟬𝟬𝟬 (𝗶𝗲 𝗜𝗡𝗥 𝟭𝟬-𝟭𝟲 𝗟𝗣𝗔)
Experience: 5+ yrs
Location: Bengaluru, Karnataka, India
Job Type: Full-time
We are looking for an experienced Senior Data QA Engineer to ensure the accuracy, reliability, and quality of enterprise data pipelines, ETL processes, and data platforms. The role focuses on validating data transformations, business logic, KPIs, metrics, and data quality across modern platforms such as Snowflake, Databricks, and Hive.
The ideal candidate will combine strong SQL, Python, data engineering, and test automation skills with a deep understanding of data quality and validation. You will work closely with Data Engineers, Data Analysts, Product Managers, and Engineering teams to identify potential issues early and ensure reliable, production-ready data releases.
Requirements
Key Responsibilities
- Validate ETL pipelines, data transformations, business logic, and data quality across Snowflake, Databricks, Hive, and other data platforms.
- Design and execute comprehensive test scenarios for data pipelines, data warehouses, and analytical datasets.
- Translate business and technical requirements into effective test cases covering KPIs, metrics, calculations, and business rules.
- Use advanced SQL to validate large datasets, identify anomalies, reconcile data, and investigate data-quality issues.
- Partner with Data Engineers to identify potential failure points and proactively detect defects before production releases.
- Develop automated and reusable tests for data pipelines to improve coverage and reduce regression risk.
- Contribute to and enhance existing data test automation frameworks with a focus on scalability, reliability, and maintainability.
- Validate data accuracy, completeness, consistency, and integrity across source, transformation, and target systems.
- Perform regression testing and release validation for data platform changes.
- Collaborate with Data Analysts, Product Managers, Data Engineers, and Engineering teams to resolve data-quality issues.
- Support testing across batch and distributed data-processing environments.
- Use Python to develop automation scripts, validation utilities, and data-quality testing solutions.
- Integrate testing practices into CI/CD workflows to improve release quality and development velocity.
- Investigate production data issues, perform root-cause analysis, and help implement sustainable solutions.
- Maintain test documentation, validation standards, and reusable testing assets.
- Continuously improve data testing methodologies, automation coverage, and quality processes.
What Makes You a Great Fit
- 5+ years of experience in data quality, data QA, ETL testing, data engineering testing, or a closely related role.
- Strong hands-on experience validating data pipelines, ETL processes, and data warehouses in production environments.
- Expert-level SQL skills with experience working with very large datasets, including terabyte-scale data.
- Proven ability to identify data anomalies, inconsistencies, and quality issues through efficient SQL analysis.
- Strong experience with Snowflake, Databricks, Hive, or similar modern data platforms.
- Solid proficiency in Python and experience developing automated tests for data pipelines.
- Good understanding of Apache Spark, Airflow, and modern data-processing workflows.
- Strong understanding of data warehousing, ETL/ELT concepts, data transformations, and data validation.
- Familiarity with CI/CD principles and integrating automated testing into development and deployment workflows.
- Experience building or contributing to scalable and maintainable test automation frameworks.
- Knowledge of BDD frameworks such as Behave is an advantage.
- Experience working with AWS or other cloud platforms is desirable.
- Familiarity with data-quality frameworks such as Great Expectations, Deequ, or similar custom solutions is an advantage.
- Strong analytical and problem-solving skills with excellent attention to detail.
- Excellent communication and collaboration skills with the ability to work effectively across technical and business teams.
- Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent professional experience is preferred.
- Strong ownership mindset with the ability to proactively identify quality risks and drive issues through resolution.
Similar Jobs
Gaming • Information Technology • Mobile • Software • Esports
Designs and maintains automated data quality checks for game telemetry, player events, and backend systems. Monitors pipelines, detects anomalies and schema changes, defines data quality standards and SLAs, investigates root causes, and builds dashboards and alerting. Collaborates with engineering, analytics, and product teams on event instrumentation, data lineage, privacy compliance, and reliable datasets for strategic decisions.
Top Skills:
A/B TestingAirflowAWSDatabricksDbtGreat ExpectationsMonte CarloSnowflakeSparkSQL
Artificial Intelligence • HR Tech • Professional Services • Software
Leads data quality engineering and automated testing across batch and streaming data pipelines. Designs scalable testing frameworks, validates data accuracy and business rules, integrates quality checks into CI/CD, troubleshoots root causes, and drives strategic quality initiatives. Partners with engineering and business teams, establishes testing standards, conducts technical reviews, mentors engineers, manages direct reports where applicable, and continuously improves data quality monitoring, coverage, and processes.
Top Skills:
Apache AirflowSparkAWSBatch ProcessingBehaveCi/CdDatabricksDeequEltETLGreat ExpectationsPythonSnowflakeSQLStreaming Data
Healthtech • Information Technology
Execute data quality and testing initiatives for a data platform, including ETL testing, data validation, load and performance testing, automated regression testing, AWS pipeline monitoring, and Power BI report validation. Develop test cases, verify ingestion and transformation logic, investigate data anomalies, document results, and collaborate with engineers, analysts, and stakeholders to maintain data integrity, availability, and reliability.
Top Skills:
AgileAmazon AthenaAmazon MskAmazon S3Amazon SqsAws GlueAws LambdaAws Step FunctionsAzure DevopsETLKanbanOopPower BIPythonRedshiftScrumSQLZephyr
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
