Lead the design and development of scalable Azure data pipelines and ETL/ELT solutions using Python, PySpark, SQL, Databricks, Data Factory, Synapse, and related Azure services. Provide technical leadership through code reviews, documentation, troubleshooting, mentoring, and engineering best practices. Collaborate with cross-functional teams to translate business requirements into technical solutions and support data warehousing, lakehouse architecture, and governance initiatives.
Lead Data EngineerRole Overview
We are looking for a Lead Data Engineer with strong experience in Azure Data Engineering, Python/PySpark, SQL, and Data Warehousing. The role involves designing scalable data pipelines, leading Azure ETL solutions, and providing technical guidance to the team.
Key Responsibilities- Design and develop scalable data pipelines using Python/PySpark.
- Architect and implement Azure ETL/ELT solutions using Databricks, Data Factory, Blob Storage, Synapse, Azure SQL, and Lakebase.
- Work with cross-functional teams to translate business requirements into technical solutions.
- Lead code reviews, ensure engineering best practices, and maintain technical documentation.
- Mentor junior engineers and provide technical leadership.
- Troubleshoot data, pipeline, and performance issues.
- Use Git, Azure DevOps, and Jira for source control and project delivery.
- 8+ years of experience in Data Engineering/Data Warehousing.
- 5+ years of experience with Python/PySpark.
- 8+ years of experience with SQL, including complex queries, stored procedures, and functions.
- Strong experience with Azure Databricks, Data Factory, Blob Storage, Synapse, Azure SQL, Lakebase, and Unity Catalog.
- Knowledge of Azure Functions, Logic Apps, Azure VMs, Git, and Azure DevOps.
- Strong problem-solving, communication, and leadership skills.
- Azure Data Engineer certification is a plus.
- Experience with Lakehouse architecture and data governance.
Role: Lead Data Engineer
Employment: Full-Time, Permanent
Experience: 8+ Years
Education: B.Tech/B.E., BCA, B.Sc., or any relevant postgraduate qualification
Similar Jobs
Fintech • Financial Services
Leads complex software engineering and AI automation initiatives. Designs, develops, tests, and deploys scalable intelligent workflows, AI agents, copilots, document-processing solutions, conversational interfaces, and RAG-based enterprise knowledge systems. Integrates AI services with enterprise applications and data sources using Microsoft Power Platform, Python, APIs, and cloud technologies. Establishes engineering standards, ensures governance and security compliance, mentors teams, and collaborates with technical and cross-functional stakeholders.
Top Skills:
Api IntegrationsAzureCi/CdDevOpsEnterprise Ai PlatformsJavaScriptLarge Language Models (Llms)Microsoft Copilot StudioMicrosoft Power PlatformPower AppsPower AutomatePythonRetrieval-Augmented Generation (Rag)ServicenowSharepointWorkday
Appliances • Manufacturing
Lead data engineering initiatives by designing and optimizing reliable data pipelines and analytics solutions. Collaborate with stakeholders, internal engineers, and external partners to prioritize projects and deliver data products. Participate in technical design reviews, improve data quality, lineage, governance, documentation, and workflows, and promote best practices across a global data engineering environment. Requires hands-on experience with Python, SQL, dbt, cloud platforms, CI/CD, and DataOps.
Top Skills:
AWSAzureCi/CdDataopsDbtGCPPythonSQL
Healthtech • Pharmaceutical • Manufacturing
Lead end-to-end enterprise data pipeline development across SAP, BigQuery, and Databricks. Own production incidents, optimize performance, implement data quality checks, and ensure CI/CD testing. Mentor junior engineers, enforce technical standards, collaborate with analysts and data scientists, document reusable patterns, and contribute to the broader data engineering community. The role focuses on scalable lakehouse data products, near-real-time replication, and reliable analytics infrastructure.
Top Skills:
SparkBigquery StudioCi/CdDatabricksDelta LakeGitGitGithub CopilotGoogle BigqueryPysparkPythonSap BwSap S/4HanaSap SltSonarqubeSpark UiSQLTest-Driven DevelopmentVs Code
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
