Design, build, test, maintain, and monitor scalable Lakeflow Spark Declarative Pipelines using Event Hubs and medallion architecture. Gather requirements, transform data across pipeline layers, troubleshoot complex issues, and create technical documentation. The role requires proficiency in Python, SQL, PySpark, Spark SQL, Databricks, Azure, and Unity Catalog, with streaming and declarative pipeline experience preferred.
Sr. Data Engineer
Data Engineer with 4+ years of experience in SQL, PySpark, ETL, ADLS, Azure Databricks, Unity Catalog, and Azure technologies.
Responsibilities- 3+ years of experience developing data pipelines using Python/PySpark, with proficiency in Spark SQL and Azure Databricks.
- 4+ years of experience with Azure data engineering technologies, including ADLS, Unity Catalog, Azure Data Factory, Azure Databricks, and Azure SQL/Synapse.
- 4+ years of experience with SQL.
- Experience in data modelling, including dimensional modelling, star schema and snowflake schemas.
- Proficiency in version control tools such as Git and project management tools such as Jira.
- Excellent verbal and written communication skills.
- UG: B.Sc, BCA, or B.Tech/B.E. in any specialization.
- Reliable internet connectivity is required.
- Experience working with financial data or finance-related datasets.
- Experience with Structured Streaming and Azure Event Hubs.
- Experience with Declarative Automation Bundles and Spark Declarative Pipelines (SDP).
- Azure certifications.
Similar Jobs
Fintech • Financial Services
Design, build, and maintain scalable batch and streaming data pipelines and data models using cloud-native technologies. Improve platform performance, reliability, observability, and automation; perform data quality validation, troubleshoot production issues, participate in architecture and code reviews, and mentor junior engineers.
Top Skills:
Apache AirflowSparkCi-CdDistributed Data ProcessingEtl/EltGitJavaJenkinsPythonSQL
Cloud • Healthtech • Professional Services • Software • Pharmaceutical
The Sr Data Engineer I will provide consulting services for clinical systems, manage projects, and design SQL code for data reporting and analytics while ensuring compliance with industry standards.
Top Skills:
.NetAWSAzureBusiness ObjectsC#CognosDb2ElluminateHTMLIbm DatastageInformaticaJreviewMicrostrategyMs SqlOraclePl/SqlPythonQlikQliksenseQlikviewRSASSparkSpotfireSQLSQL ServerSsrsT-SqlTableauTeradata
Cloud • Healthtech • Professional Services • Software • Pharmaceutical
The Senior Software Developer will develop and support Qlik Sense applications, ensure best practices, and improve analytics processes while collaborating with clients and teams.
Top Skills:
ETLQlik SenseSQL Server
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
