Design, build, and operate high-throughput, low-latency Kafka-based streaming architectures and data pipelines using Flink, Java, and PySpark. Manage and tune Kafka clusters, implement HA and disaster recovery, apply security (SSL/mTLS, SASL, ACLs), automate deployments with IaC (Terraform) and GitOps, and collaborate with cross-functional teams to deliver enterprise-grade real-time data solutions.
We are looking for an experienced Senior Data Engineer with strong expertise in real-time streaming technologies and large-scale data engineering solutions. The ideal candidate will have hands-on experience designing, building, and managing highly scalable and fault-tolerant streaming data platforms using Kafka, Flink, Java, and PySpark.
The candidate will be responsible for developing high-throughput, low-latency data pipelines, managing Kafka clusters, implementing cloud-native data solutions, and ensuring system reliability, security, and scalability in enterprise environments.
RequirementsKey Responsibilities
- Design, develop, implement, and manage Kafka-based real-time streaming architectures capable of handling high-volume and low-latency workloads.
- Build and maintain scalable streaming data pipelines using Kafka ecosystem components, including:
- Kafka Connect
- ksqlDB
- Schema Registry
- Develop and optimize real-time data processing applications using:
- Apache Flink
- Java
- PySpark
- Perform Kafka cluster setup, administration, configuration, tuning, monitoring, and performance optimization.
- Ensure Kafka clusters are highly available and implement disaster recovery strategies and best practices.
- Design and implement fault-tolerant, scalable, and resilient streaming systems for enterprise-grade applications.
- Work with cloud-native applications and modern data engineering frameworks to build scalable solutions.
- Automate infrastructure provisioning and deployment using Infrastructure-as-Code (IaC) tools such as Terraform.
- Implement and follow GitOps practices for deployment automation and infrastructure management.
- Apply robust security standards and best practices, including:
- SSL / mTLS
- SASL authentication
- ACL management
- Collaborate with cross-functional teams to deliver real-time data engineering solutions aligned with business and analytics requirements.
- Apache Kafka
- Apache Flink
- Java
- PySpark
- Real-Time Streaming Architecture
- Kafka Cluster Management
- Kafka Connect
- ksqlDB
- Schema Registry
- Streaming Data Pipelines
- Cloud-Native Applications
- Terraform
- Infrastructure-as-Code (IaC)
- GitOps
- High Availability & Disaster Recovery
- Performance Tuning & Optimization
- Fault Tolerance & Scalability
- SSL / mTLS
- SASL
- ACLs
- Strong experience in building enterprise-grade real-time data platforms.
- Hands-on experience working in high-throughput and low-latency environments.
- Experience with large-scale distributed data processing systems.
- Exposure to cloud platforms and modern DevOps practices will be an added advantage.
Similar Jobs
.png){kind=logo}
AdTech • eCommerce • Information Technology • Software • Travel • Generative AI
Design, build, and operate scalable batch and streaming data pipelines, analytics layers, and impression data models to support ad inventory forecasting. Ensure data quality, observability, and performance; collaborate with engineers, data scientists, and product teams to translate requirements into reliable, production-grade data solutions and integrate AI/ML capabilities where appropriate.
Top Skills:
Ai/MlAlertingApi DesignCi/CdData LakeData ModelingData WarehouseDeployment AutomationDistributed ProcessingFeature PipelinesJavaMonitoringObservabilityPythonScalaSQLStream Processing
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Perform and review fund accounting for Luxembourg real estate funds, calculate NAVs, fees and investor allocations, manage SPV consolidations, support audits, investor services and regulatory compliance (AIFMD/CSSF). Liaise with auditors, valuers, property managers and cross-functional teams, improve processes, and mentor junior staff.
Cloud • Security • Software • Cybersecurity • Automation
Design, build, and deploy enterprise UiPath RPA automations end-to-end. Gather requirements, create reusable solutions, document designs, support production bots, perform root-cause analysis, and continuously optimize automations while collaborating with stakeholders and Enterprise Apps teams.
Top Skills:
AdpCi/CdCoupaDevOpsEtradeJSONNavanNetSuiteOauthPythonRestSalesforceSAMLSoapSoxSQLUipath Agentic AiUipath Automation HubUipath OrchestratorUipath RobotsUipath StudioWorkdayXMLZipZuora BillingZuora Revenue
What you need to know about the Bengaluru Tech Scene
Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.
