Design, build, and maintain scalable Azure-based data pipelines, lakes, and warehouses (ADF, Databricks, ADLS, Synapse). Implement ETL/ELT for batch and real-time ingestion, ensure data quality, security, and governance, optimize performance and cost, support production pipelines, enable analytics, and implement CI/CD and documentation.
Data Engineer
Job Summary
We are seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data solutions on Microsoft Azure. The ideal candidate will have strong expertise in data engineering, ETL/ELT development, cloud technologies, and big data processing frameworks. This role involves building and optimizing data pipelines, integrating data from multiple sources, and enabling analytics and reporting across the organization.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and data integration solutions using Azure services.
- Build and optimize ETL/ELT workflows using Azure Data Factory (ADF), Azure Databricks, and other Azure-native tools.
- Develop and manage data lakes and data warehouses using Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL.
- Implement data ingestion frameworks for batch and real-time data processing.
- Work with structured and unstructured datasets from various internal and external sources.
- Ensure data quality, integrity, security, and governance across the data platform.
- Optimize data processing jobs for performance, scalability, and cost efficiency.
- Collaborate with business analysts, data scientists, architects, and application teams to understand data requirements.
- Monitor, troubleshoot, and support production data pipelines.
- Implement CI/CD processes and DevOps best practices for data engineering solutions.
- Create and maintain technical documentation, data lineage, and operational procedures.
Required Skills & Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- 3+ years of experience in data engineering or a related role.
- Strong experience with Microsoft Azure cloud services.
- Hands-on experience with:
- Databricks
- Azure Data Lake Storage (ADLS Gen2)
- Azure SQL Database
- Strong SQL development and query optimization skills.
- Experience with Python, PySpark.
- Knowledge of data modeling, data warehousing, and dimensional modeling concepts.
- Understanding of data security, governance, and access control principles.
- Familiarity with Agile development methodologies.
Preferred Qualifications
- Experience with Delta Lake and Medallion Architecture.
- Knowledge of streaming technologies such as Azure Event Hub or Azure Stream Analytics.
- Experience with Power BI integration and analytics solutions.
- Azure certifications such as:
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
Technical Skills
- Azure Data Factory (ADF)
- Azure Databricks
- PySpark
- SQL
- Python
- Delta Lake
- Azure DevOps
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management
- Ability to work independently and within cross-functional teams
- Strong attention to detail and commitment to data quality
Nice to Have
- Experience with Kafka or Event Hub
- Experience with Databricks Applications
- Exposure to machine learning data pipelines
- Experience in retail, finance, healthcare, or e-commerce domains
Similar Jobs
.png){kind=logo}
AdTech • eCommerce • Information Technology • Software • Travel • Generative AI
Design, build, and operate scalable batch and streaming data pipelines, analytics layers, and impression data models to support ad inventory forecasting. Ensure data quality, observability, and performance; collaborate with engineers, data scientists, and product teams to translate requirements into reliable, production-grade data solutions and integrate AI/ML capabilities where appropriate.
Top Skills:
Ai/MlAlertingApi DesignCi/CdData LakeData ModelingData WarehouseDeployment AutomationDistributed ProcessingFeature PipelinesJavaMonitoringObservabilityPythonScalaSQLStream Processing
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Perform and review fund accounting for Luxembourg real estate funds, calculate NAVs, fees and investor allocations, manage SPV consolidations, support audits, investor services and regulatory compliance (AIFMD/CSSF). Liaise with auditors, valuers, property managers and cross-functional teams, improve processes, and mentor junior staff.
Cloud • Security • Software • Cybersecurity • Automation
Design, build, and deploy enterprise UiPath RPA automations end-to-end. Gather requirements, create reusable solutions, document designs, support production bots, perform root-cause analysis, and continuously optimize automations while collaborating with stakeholders and Enterprise Apps teams.
Top Skills:
AdpCi/CdCoupaDevOpsEtradeJSONNavanNetSuiteOauthPythonRestSalesforceSAMLSoapSoxSQLUipath Agentic AiUipath Automation HubUipath OrchestratorUipath RobotsUipath StudioWorkdayXMLZipZuora BillingZuora Revenue
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
