Design, build, and maintain scalable Azure-based data pipelines, lakes, and warehouses (ADF, Databricks, ADLS, Synapse). Implement ETL/ELT for batch and real-time ingestion, ensure data quality, security, and governance, optimize performance and cost, support production pipelines, enable analytics, and implement CI/CD and documentation.
Data Engineer
Job Summary
We are seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data solutions on Microsoft Azure. The ideal candidate will have strong expertise in data engineering, ETL/ELT development, cloud technologies, and big data processing frameworks. This role involves building and optimizing data pipelines, integrating data from multiple sources, and enabling analytics and reporting across the organization.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and data integration solutions using Azure services.
- Build and optimize ETL/ELT workflows using Azure Data Factory (ADF), Azure Databricks, and other Azure-native tools.
- Develop and manage data lakes and data warehouses using Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL.
- Implement data ingestion frameworks for batch and real-time data processing.
- Work with structured and unstructured datasets from various internal and external sources.
- Ensure data quality, integrity, security, and governance across the data platform.
- Optimize data processing jobs for performance, scalability, and cost efficiency.
- Collaborate with business analysts, data scientists, architects, and application teams to understand data requirements.
- Monitor, troubleshoot, and support production data pipelines.
- Implement CI/CD processes and DevOps best practices for data engineering solutions.
- Create and maintain technical documentation, data lineage, and operational procedures.
Required Skills & Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- 3+ years of experience in data engineering or a related role.
- Strong experience with Microsoft Azure cloud services.
- Hands-on experience with:
- Databricks
- Azure Data Lake Storage (ADLS Gen2)
- Azure SQL Database
- Strong SQL development and query optimization skills.
- Experience with Python, PySpark.
- Knowledge of data modeling, data warehousing, and dimensional modeling concepts.
- Understanding of data security, governance, and access control principles.
- Familiarity with Agile development methodologies.
Preferred Qualifications
- Experience with Delta Lake and Medallion Architecture.
- Knowledge of streaming technologies such as Azure Event Hub or Azure Stream Analytics.
- Experience with Power BI integration and analytics solutions.
- Azure certifications such as:
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
Technical Skills
- Azure Data Factory (ADF)
- Azure Databricks
- PySpark
- SQL
- Python
- Delta Lake
- Azure DevOps
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management
- Ability to work independently and within cross-functional teams
- Strong attention to detail and commitment to data quality
Nice to Have
- Experience with Kafka or Event Hub
- Experience with Databricks Applications
- Exposure to machine learning data pipelines
- Experience in retail, finance, healthcare, or e-commerce domains
Similar Jobs
.png){kind=logo}
Food • Greentech • Logistics • Sharing Economy • Transportation • Agriculture • Industrial
Collects, processes, and analyzes complex datasets; builds automated reports and dashboards; applies analytics and AI/ML techniques; automates data workflows and low-code apps; ensures data quality; partners with stakeholders and engineering teams to deliver actionable insights across the Asia Pacific region.
Top Skills:
AWSExcelPower AppsPower AutomatePower BIPower QueryPythonSnowflakeSQLVBA
Food • Greentech • Logistics • Sharing Economy • Transportation • Agriculture • Industrial
Design and implement scalable enterprise Salesforce solutions, lead architecture and design reviews, build reusable components, integrate Salesforce with enterprise and AI systems, mentor engineers, and ensure security, scalability, and governance.
Top Skills:
AgentforceAi AgentsAnthropicApexAutomated TestingAws BedrockAzure OpenaiCi/CdData CloudEinstein AiEvent-Driven ArchitectureGeminiGitLightning Web ComponentsMcpMicroservicesModel BuilderMulesoftOpenaiPlatform EventsPrompt BuilderRagRest ApisSalesforceSalesforce ApisSalesforce FlowsSOQLSoslStreaming ApisVector Databases
Fintech • Machine Learning • Payments • Software • Financial Services
Lead AML-focused business analysis and product delivery: develop strategy, translate analytics into products, manage multiple projects, partner with Tech/Model Risk, mentor analysts, and communicate results to senior stakeholders.
Top Skills:
AWSMachine LearningPythonRRelational DatabasesSnowflakeSparkSQL
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
