The Senior Data Engineer will design and optimize scalable data pipelines, implement real-time streaming pipelines, collaborate with data scientists, and oversee CI/CD and data quality practices.
We are looking for a highly skilled Senior Data Engineer to join our Data Engineering chapter within the Technology Platform domain. The ideal candidate will have strong expertise in real-time data streaming, modern data platforms, and AI/ML operationalization (AIOps/MLOps/LLMOps).
You will be responsible for designing, building, and optimizing scalable data pipelines and enabling advanced analytics and AI-driven solutions.
RequirementsKey Responsibilities
- Design and implement real-time data streaming pipelines using Kafka and related technologies
- Build and maintain scalable data engineering solutions on Cloudera platform
- Develop and optimize ETL/ELT pipelines for large-scale structured and unstructured data
- Collaborate with data scientists to deploy and manage ML models (MLOps/LLMOps frameworks)
- Implement and manage CI/CD pipelines for data and ML workflows
- Work with OpenShift/Kubernetes for containerized deployments
- Ensure data quality, governance, and security best practices
- Monitor, troubleshoot, and optimize data workflows and infrastructure
- Partner with cross-functional teams including product, engineering, and analytics
- 6+ years of experience in Data Engineering
- Strong hands-on experience with Apache Kafka (Real-Time Streaming - RTS)
- Expertise in Cloudera Data Platform (CDP) or similar big data ecosystems
- Experience with CI/CD pipelines (Jenkins, GitLab, etc.)
- Hands-on experience with OpenShift/Kubernetes
- Strong programming skills in Python/Scala/Java
- Experience with distributed data processing frameworks (Spark, Hive, etc.)
- Good understanding of data modeling and data warehousing concepts
- Experience in AIOps, MLOps, or LLMOps
- Exposure to AI/ML lifecycle management and deployment
- Knowledge of cloud platforms (AWS/Azure/GCP) is a plus
- Experience in banking or financial services domain
- Strong problem-solving and analytical thinking
- Excellent communication and stakeholder management
- Ability to work in agile and fast-paced environments
Similar Jobs
Artificial Intelligence • Software • Automation
The Senior Machine Learning Engineer will develop AI-assisted annotation pipelines, leverage large models for training data, and validate outputs for model training. Responsibilities include overseeing the automated annotation process, optimizing inference pipelines, and collaborating with various teams to enhance annotation quality and system performance.
Top Skills:
PythonPyTorch
Artificial Intelligence • Software • Consulting • Generative AI
The Sr. Data Scientist / AI/ML Engineer will design, develop, and implement AI solutions, develop scalable AI models, and mentor junior engineers while collaborating on diverse AI projects.
Top Skills:
AIAWSAws SagemakerAzureDockerGCPJenkinsKubernetesMachine LearningPyTorchSeleniumTensorFlowTestng
Other
As a Data/ML Engineer, you will design and implement data pipelines, optimize workflows for unstructured data, and integrate LLM frameworks to power analytics and BI solutions.
Top Skills:
SparkAWSAzure Machine LearningHadoopHugging Face TransformersKafkaLangchainLlamaindexPlsqlPythonRedshiftSnowflakeSQL
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
