Design, build, and maintain scalable Azure-based data pipelines, lakes, and warehouses (ADF, Databricks, ADLS, Synapse). Implement ETL/ELT for batch and real-time ingestion, ensure data quality, security, and governance, optimize performance and cost, support production pipelines, enable analytics, and implement CI/CD and documentation.
Data Engineer
Job Summary
We are seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data solutions on Microsoft Azure. The ideal candidate will have strong expertise in data engineering, ETL/ELT development, cloud technologies, and big data processing frameworks. This role involves building and optimizing data pipelines, integrating data from multiple sources, and enabling analytics and reporting across the organization.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and data integration solutions using Azure services.
- Build and optimize ETL/ELT workflows using Azure Data Factory (ADF), Azure Databricks, and other Azure-native tools.
- Develop and manage data lakes and data warehouses using Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL.
- Implement data ingestion frameworks for batch and real-time data processing.
- Work with structured and unstructured datasets from various internal and external sources.
- Ensure data quality, integrity, security, and governance across the data platform.
- Optimize data processing jobs for performance, scalability, and cost efficiency.
- Collaborate with business analysts, data scientists, architects, and application teams to understand data requirements.
- Monitor, troubleshoot, and support production data pipelines.
- Implement CI/CD processes and DevOps best practices for data engineering solutions.
- Create and maintain technical documentation, data lineage, and operational procedures.
Required Skills & Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- 3+ years of experience in data engineering or a related role.
- Strong experience with Microsoft Azure cloud services.
- Hands-on experience with:
- Databricks
- Azure Data Lake Storage (ADLS Gen2)
- Azure SQL Database
- Strong SQL development and query optimization skills.
- Experience with Python, PySpark.
- Knowledge of data modeling, data warehousing, and dimensional modeling concepts.
- Understanding of data security, governance, and access control principles.
- Familiarity with Agile development methodologies.
Preferred Qualifications
- Experience with Delta Lake and Medallion Architecture.
- Knowledge of streaming technologies such as Azure Event Hub or Azure Stream Analytics.
- Experience with Power BI integration and analytics solutions.
- Azure certifications such as:
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
Technical Skills
- Azure Data Factory (ADF)
- Azure Databricks
- PySpark
- SQL
- Python
- Delta Lake
- Azure DevOps
Soft Skills
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management
- Ability to work independently and within cross-functional teams
- Strong attention to detail and commitment to data quality
Nice to Have
- Experience with Kafka or Event Hub
- Experience with Databricks Applications
- Exposure to machine learning data pipelines
- Experience in retail, finance, healthcare, or e-commerce domains
Similar Jobs
Cloud • Information Technology • Security • Software
Own and scale a $1M territory through partner recruitment, enablement, and GTM execution. Build executive partner relationships, manage co-selling and pipeline, run performance analysis, and coordinate internal teams to drive channel revenue in India.
Top Skills:
Active DirectoryGeminiGongGoogle Workspace (Gws)ImpartnerJumpcloudSalesforce (Sfdc)
Artificial Intelligence • Natural Language Processing • Professional Services • Analytics • Consulting • Conversational AI • Generative AI
Lead module functional consultant for end-to-end Oracle HCM Cloud (Fusion HCM) implementations. Must have functional/configuration experience in two or more modules such as Core HR, Absence, US/Canada Payroll, Benefits, Compensation, Talent Management, ORC, Learning, OTL, or HR Helpdesk.
Top Skills:
AbsenceBenefitsCanada PayrollCompensationCore HrFusion HcmHr HelpdeskLearningOracle Fusion HcmOracle Hcm CloudOrcOtlTalent ManagementUs Payroll
Artificial Intelligence • Natural Language Processing • Professional Services • Analytics • Consulting • Conversational AI • Generative AI
Lead design and architecture of modern data ecosystems (data lake/lakehouse), data modeling, ETL, and API-based architectures. Build RESTful/GraphQL and event-driven APIs, API gateway/auth patterns, and agentic AI agents/workflows. Provide solution leadership, stakeholder engagement, governance, and cloud data platform (preferably AWS) expertise.
Top Skills:
Agentic AiApi GatewayApi VersioningAuthentication/AuthorizationAWSData LakeData ModelingETLEvent-Driven ArchitectureGraphQLLakehouseRate LimitingRestful Api
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
