Lead the design and development of scalable Azure data pipelines and ETL/ELT solutions using Python, PySpark, SQL, Databricks, Data Factory, Synapse, and related Azure services. Provide technical leadership through code reviews, documentation, troubleshooting, mentoring, and engineering best practices. Collaborate with cross-functional teams to translate business requirements into technical solutions and support data warehousing, lakehouse architecture, and governance initiatives.
Lead Data EngineerRole Overview
We are looking for a Lead Data Engineer with strong experience in Azure Data Engineering, Python/PySpark, SQL, and Data Warehousing. The role involves designing scalable data pipelines, leading Azure ETL solutions, and providing technical guidance to the team.
Key Responsibilities- Design and develop scalable data pipelines using Python/PySpark.
- Architect and implement Azure ETL/ELT solutions using Databricks, Data Factory, Blob Storage, Synapse, Azure SQL, and Lakebase.
- Work with cross-functional teams to translate business requirements into technical solutions.
- Lead code reviews, ensure engineering best practices, and maintain technical documentation.
- Mentor junior engineers and provide technical leadership.
- Troubleshoot data, pipeline, and performance issues.
- Use Git, Azure DevOps, and Jira for source control and project delivery.
- 8+ years of experience in Data Engineering/Data Warehousing.
- 5+ years of experience with Python/PySpark.
- 8+ years of experience with SQL, including complex queries, stored procedures, and functions.
- Strong experience with Azure Databricks, Data Factory, Blob Storage, Synapse, Azure SQL, Lakebase, and Unity Catalog.
- Knowledge of Azure Functions, Logic Apps, Azure VMs, Git, and Azure DevOps.
- Strong problem-solving, communication, and leadership skills.
- Azure Data Engineer certification is a plus.
- Experience with Lakehouse architecture and data governance.
Role: Lead Data Engineer
Employment: Full-Time, Permanent
Experience: 8+ Years
Education: B.Tech/B.E., BCA, B.Sc., or any relevant postgraduate qualification
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Design, develop, and maintain scalable web applications, APIs, microservices, distributed systems, and real-time data solutions across hybrid on-premises and Azure environments. Translate product requirements into technical designs, modernize enterprise applications, troubleshoot production issues, and drive engineering quality through testing, monitoring, observability, and code reviews. Collaborate cross-functionally, support business-critical applications, integrate AI technologies, and contribute across the full software development lifecycle.
Top Skills:
AzureAzure DevopsClaude CodeCursorDockerEvent HubsGithub ActionsGithub CopilotJavaJenkinsKafkaKubernetesLlmsMicroservicesNosql DatabasesRabbitMQRagRelational DatabasesRest Apis
Information Technology
Lead data engineering role responsible for developing and optimizing dbt-based ETL/ELT pipelines, administering Snowflake warehouses, managing AWS cloud infrastructure, and implementing CI/CD, monitoring, governance, and data quality controls. The role includes troubleshooting production issues, supporting analytics consumers, maintaining APIs, documenting data lineage, and collaborating with engineering, BI, analytics, and governance teams.
Top Skills:
Aws CloudwatchAws IamAws LambdaAws S3Aws Secrets ManagerCi/CdDbtEtl/EltPythonSnowflakeSQL
Edtech • HR Tech • Information Technology • Professional Services
Lead the design and management of scalable, AI-ready data platforms and products. Build batch and real-time pipelines, semantic layers, and data models while ensuring governance, quality, reliability, and SLA adherence. Enable ML and agentic AI capabilities, implement DataOps and CI/CD, optimize cloud costs, and maintain platform stability. Collaborate with stakeholders to translate business needs into sustainable technical solutions.
Top Skills:
Agentic AiApache AirflowSparkAPIsAstronomerAWSAzureBigQueryCi/CdData GovernanceData ModelingDatabricksDataflowDataopsEvent HubGCPKafkaKinesisLlm IntegrationsMetadata ManagementMicroservicesPower BIPysparkPythonReactSnowflakeSQLSynapse
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
