The Senior Data Engineer will design, build, and manage data pipelines using Azure services while ensuring data quality and compliance. This role involves collaboration with cross-functional teams and mentoring junior engineers.
Job Overview:
We are seeking a highly skilled Senior Data Engineer with deep expertise in Microsoft Azure data ecosystem to design, develop, and maintain scalable data pipelines and architectures. The ideal candidate will play a key role in building robust data solutions that support advanced analytics, BI, and AI workloads across the organization. This role involves working with cross-functional teams — data scientists, analysts, and business stakeholders — to ensure high-quality, secure, and performant data delivery.
Total experience : 8 to 10 years
Key Responsibilities:
1. Data Architecture & Pipeline Development
- Design, develop, and maintain data ingestion, transformation, and storage pipelines using Azure Data Factory, Azure Databricks, and Synapse Analytics.
- Build end-to-end ETL/ELT workflows from diverse sources such as APIs, on-prem databases, SaaS applications, and data lakes.
- Implement data modeling (star schema, snowflake, data vault, etc.) for analytical and operational data stores.
- Manage data ingestion frameworks for both batch and streaming (real-time) use cases using Azure Event Hubs / Azure Stream Analytics / Kafka.
2. Data Management & Governance
- Ensure data quality, consistency, and integrity across all environments.
- Implement and enforce data governance standards, including metadata management, lineage tracking, and data cataloging (e.g., Azure Purview).
- Manage security, compliance, and access controls using Azure Active Directory (AAD) and RBAC principles.
- Automate data validation, auditing, and monitoring workflows using modern DevOps practices.
3. Cloud & DevOps Integration
- Manage and optimize Azure Data Lake Storage (ADLS Gen2) for cost and performance.
- Leverage Infrastructure as Code (IaC) tools such as Terraform, Bicep, or ARM templates for environment provisioning.
- Implement CI/CD pipelines for data workflows using Azure DevOps / GitHub Actions.
- Optimize compute and storage costs while maintaining scalability and resilience.
4. Collaboration & Leadership
- Partner with data scientists, BI developers, and product teams to deliver reliable data solutions.
- Mentor junior data engineers on best practices, coding standards, and Azure services.
- Participate in architectural reviews and contribute to data platform design decisions.
- Communicate technical insights and trade-offs to non-technical stakeholders effectively.
Core Azure Services:
- Azure Data Factory (ADF) – pipeline orchestration, triggers, linked services, integration runtime.
- Azure Databricks – PySpark, Delta Lake, notebooks, job orchestration.
- Azure Synapse Analytics – dedicated and serverless SQL pools, data modeling, query optimization.
- Azure Data Lake Storage (ADLS Gen2) – hierarchical namespace, lifecycle management.
- Azure Functions / Logic Apps – event-driven data workflows.
- Azure Event Hubs / Kafka / Stream Analytics – real-time data ingestion.
Programming & Tools:
- Strong in Python or Scala for data transformations.
- Proficient in SQL (T-SQL, Spark SQL, or Synapse SQL).
- Experience with PySpark for distributed data processing.
- Knowledge of Git, CI/CD, and DevOps best practices.
- Familiarity with Power BI integration or other reporting tools is a plus.
Data Architecture & Modeling:
- Expertise in data warehouse design, data lakehouse architecture, and data pipelines.
- Understanding of data partitioning, indexing, and query optimization techniques.
- Experience with dimensional modeling, slowly changing dimensions, and fact tables.
Other Skills:
- Experience with API-based data ingestion and RESTful integrations.
- Exposure to machine learning data pipelines or feature stores is a plus.
- Familiarity with cost optimization and performance tuning in Azure.
Soft Skills
- Excellent analytical and problem-solving abilities.
- Strong communication and documentation skills.
- Ability to work in an agile, fast-paced environment.
- Proven experience collaborating in cross-functional teams.
- Self-driven with a passion for data engineering and cloud technologies.
Top Skills
Arm Templates
Azure Active Directory
Azure Data Factory
Azure Databricks
Azure Devops
Azure Event Hubs
Azure Synapse Analytics
Bicep
Kafka
Azure
Power BI
Pyspark
Python
Scala
SQL
Terraform
Similar Jobs
Financial Services
Lead Software Engineer at JPMorgan Chase responsible for enhancing technology products, providing technical guidance, developing secure code, and serving as a subject matter expert. Requires deep technical expertise and problem-solving skills over multiple technologies.
Top Skills:
AWSJavaNoSQLSpringSQL
Fintech • Financial Services
The role involves leading a team to drive HR transformation through automation solutions, collaborating with various stakeholders, and ensuring compliance with regulations.
Top Skills:
AIAlfrescoPower AutomateRpaServicenowUipathWorkday
Fintech • Financial Services
The Senior Risk Analytics Consultant leads analytics initiatives for consumer lending, ensuring data accuracy and risk reporting while collaborating with teams to drive business insights.
Top Skills:
AutosysPythonSASSas ViyaSQLSQL ServerTableauTeradataUnix Scripting
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
