Build and maintain scalable consumer-facing data pipelines on Databricks using PySpark, Spark SQL, and Delta Lake. Implement medallion architecture, Unity Catalog, and Workflows. Optimize cluster performance and costs, and monitor and troubleshoot production jobs. The role requires data engineering experience with Databricks, PySpark, SQL, Git, and CI/CD; pharma or consumer data experience and Databricks certification are preferred.
Build and maintain scalable data pipelines on Databricks for consumer-facing IBU data products.
- Develop ETL/ELT pipelines using PySpark, Spark SQL and Delta Lake
- Implement medallion architecture (bronze/silver/gold), Unity Catalog and Workflows
- Optimize cluster performance and cost; monitor and troubleshoot jobs in production
Requirements
- Must have: 4+ yrs data engineering, Databricks, PySpark, SQL, Git/CI-CD.
- Nice to have: pharma/consumer data, Databricks certification
Similar Jobs
Fintech • Financial Services
Provides enterprise technical leadership for complex engineering challenges across networking, application delivery, information security, databases, operating systems, and web technologies. Defines architecture and engineering standards, leads hybrid-cloud modernization, and designs highly available, scalable, secure infrastructure using F5 BIG-IP and AVI load balancers. Drives automation, Infrastructure as Code, DevOps, resiliency, production support, and incident resolution while advising leadership, influencing strategy, and mentoring engineers.
Top Skills:
AnsibleAvi Load BalancerAWSAzureCi/CdCiscoDnsF5 Big-IpGCPGtmKubernetesLtmOpenshiftPythonSite Reliability EngineeringSsl/TlsTcp/IpTerraformVMwareWafZero Trust
Fintech • Financial Services
Owns product demand, roadmaps, backlogs, requirements, Agile delivery, testing, release readiness, governance, and executive reporting across market maker and options clearing. Coordinates Product, Technology, Operations, Risk, Compliance, Finance, and Control stakeholders while managing dependencies, risks, controls, regulatory initiatives, and process improvements. Supports automation, analytics, AI enablement, production governance, and operational readiness for clearing and post-trade capabilities.
Top Skills:
Agile Delivery MethodologiesAi-Enabled ToolsAnalyticsJIRALow-Code PlatformsExcelMS OfficeMicrosoft PowerpointMicrosoft WordWorkflow Automation
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Principal Software Engineer focused on encryption, working remotely within Atlassian’s distributed-first engineering organization. The provided description contains limited role-specific responsibilities.
Top Skills:
Encryption
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
