The Senior Data Engineer will design and optimize scalable data pipelines, implement real-time streaming pipelines, collaborate with data scientists, and oversee CI/CD and data quality practices.
We are looking for a highly skilled Senior Data Engineer to join our Data Engineering chapter within the Technology Platform domain. The ideal candidate will have strong expertise in real-time data streaming, modern data platforms, and AI/ML operationalization (AIOps/MLOps/LLMOps).
You will be responsible for designing, building, and optimizing scalable data pipelines and enabling advanced analytics and AI-driven solutions.
RequirementsKey Responsibilities
- Design and implement real-time data streaming pipelines using Kafka and related technologies
- Build and maintain scalable data engineering solutions on Cloudera platform
- Develop and optimize ETL/ELT pipelines for large-scale structured and unstructured data
- Collaborate with data scientists to deploy and manage ML models (MLOps/LLMOps frameworks)
- Implement and manage CI/CD pipelines for data and ML workflows
- Work with OpenShift/Kubernetes for containerized deployments
- Ensure data quality, governance, and security best practices
- Monitor, troubleshoot, and optimize data workflows and infrastructure
- Partner with cross-functional teams including product, engineering, and analytics
- 6+ years of experience in Data Engineering
- Strong hands-on experience with Apache Kafka (Real-Time Streaming - RTS)
- Expertise in Cloudera Data Platform (CDP) or similar big data ecosystems
- Experience with CI/CD pipelines (Jenkins, GitLab, etc.)
- Hands-on experience with OpenShift/Kubernetes
- Strong programming skills in Python/Scala/Java
- Experience with distributed data processing frameworks (Spark, Hive, etc.)
- Good understanding of data modeling and data warehousing concepts
- Experience in AIOps, MLOps, or LLMOps
- Exposure to AI/ML lifecycle management and deployment
- Knowledge of cloud platforms (AWS/Azure/GCP) is a plus
- Experience in banking or financial services domain
- Strong problem-solving and analytical thinking
- Excellent communication and stakeholder management
- Ability to work in agile and fast-paced environments
Similar Jobs
Artificial Intelligence • Big Data • Cloud • Software
As a Consultant, you will partner with customers, gather functional requirements, provide recommendations, ensure delivery quality, and handle documentation. Responsibilities include managing projects, engaging with stakeholders, and supporting agile methods for implementation.
Top Skills:
JIRAProsciPythonR
Artificial Intelligence • Big Data • Cloud • Software
The Program Manager will oversee B2B customer implementation projects, ensuring they are delivered on time and within budget while managing stakeholder relationships and coordinating cross-functional teams.
Top Skills:
JIRAMs ProjectSmartsheet
Software
As a Backend Software Engineer at Postman, you will implement features, improve performance, conduct code reviews, and enhance integration with developer tools.
Top Skills:
GoNode.jsPythonRuby
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
