The Senior Data Engineer will design and optimize scalable data pipelines, implement real-time streaming pipelines, collaborate with data scientists, and oversee CI/CD and data quality practices.
We are looking for a highly skilled Senior Data Engineer to join our Data Engineering chapter within the Technology Platform domain. The ideal candidate will have strong expertise in real-time data streaming, modern data platforms, and AI/ML operationalization (AIOps/MLOps/LLMOps).
You will be responsible for designing, building, and optimizing scalable data pipelines and enabling advanced analytics and AI-driven solutions.
RequirementsKey Responsibilities
- Design and implement real-time data streaming pipelines using Kafka and related technologies
- Build and maintain scalable data engineering solutions on Cloudera platform
- Develop and optimize ETL/ELT pipelines for large-scale structured and unstructured data
- Collaborate with data scientists to deploy and manage ML models (MLOps/LLMOps frameworks)
- Implement and manage CI/CD pipelines for data and ML workflows
- Work with OpenShift/Kubernetes for containerized deployments
- Ensure data quality, governance, and security best practices
- Monitor, troubleshoot, and optimize data workflows and infrastructure
- Partner with cross-functional teams including product, engineering, and analytics
- 6+ years of experience in Data Engineering
- Strong hands-on experience with Apache Kafka (Real-Time Streaming - RTS)
- Expertise in Cloudera Data Platform (CDP) or similar big data ecosystems
- Experience with CI/CD pipelines (Jenkins, GitLab, etc.)
- Hands-on experience with OpenShift/Kubernetes
- Strong programming skills in Python/Scala/Java
- Experience with distributed data processing frameworks (Spark, Hive, etc.)
- Good understanding of data modeling and data warehousing concepts
- Experience in AIOps, MLOps, or LLMOps
- Exposure to AI/ML lifecycle management and deployment
- Knowledge of cloud platforms (AWS/Azure/GCP) is a plus
- Experience in banking or financial services domain
- Strong problem-solving and analytical thinking
- Excellent communication and stakeholder management
- Ability to work in agile and fast-paced environments
Top Skills
AWS
Azure
Ci/Cd
Cloudera
GCP
Hive
Java
Kafka
Kubernetes
Openshift
Python
Scala
Spark
Similar Jobs
Cloud • Information Technology • Internet of Things • Machine Learning • Software • Cybersecurity • Infrastructure as a Service (IaaS)
The Senior Engineer will verify ASIC designs, implement UVM test environments, develop and run test cases, and collaborate across teams to enhance verification methodologies.
Top Skills:
AmbaAxiCadenceChiJenkinsJIRASimscopeSynopsysSystemverilogUvm
Cloud • Information Technology • Security • Software • Cybersecurity
The role involves overseeing identity and access control infrastructure, leading a team in Zero Trust technology, managing development, and collaborating with various stakeholders to ensure efficient project delivery.
Top Skills:
AWSAzureCassandraDynamoDBGCPGoogle SpannerKafkaKubernetesLdapMemcachedOauthRabbitMQRedisSAMLScim
Cloud • Information Technology • Security • Software • Cybersecurity
As a Staff Software Development Engineer, you will lead the development of identity management products, design core functionalities, and collaborate with teams to ensure excellence in software delivery.
Top Skills:
Ci/CdDockerHibernateJavaKubernetesPostgresRestful ApisSpringSQL
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
