Yadimen Consulting Limited Logo

Yadimen Consulting Limited

Data Engineer — PySpark + AWS Glue

Posted 16 Days Ago
Be an Early Applicant
In-Office
Bangalore, Bengaluru Urban, Karnataka, IND
Senior level
In-Office
Bangalore, Bengaluru Urban, Karnataka, IND
Senior level
Design, develop, optimize, and maintain scalable ETL pipelines using PySpark and AWS Glue. Build data ingestion and transformation workflows, orchestrate processes with AWS Step Functions, and develop Python-based AWS Lambda functions. Support AWS data lake architectures, analytical use cases, monitoring, reliability, and performance optimization. The role also involves SQL and may include Java microservices, REST APIs, and backend integrations.
The summary above was generated by AI

Data Engineer — PySpark + AWS Glue

ETL & Data Pipelines | 2 Openings

 

 

Location: Remote (UK)

Employment Type: Contract / Permanent

Experience Level: 4–8 Years

Openings: 2

 

About the Role

We are looking for a skilled Data Engineer with solid hands-on experience in PySpark, AWS Glue, and ETL development to build and maintain scalable, production-grade data pipelines on AWS. You will be part of a delivery-focused team working on complex data engineering challenges, contributing to data lake architecture and end-to-end pipeline development.



Requirements

Key Responsibilities

       Design, develop, and maintain scalable ETL pipelines using PySpark and AWS Glue

       Build and optimise data ingestion, transformation, and loading workflows

       Orchestrate data workflows using AWS Step Functions

       Develop serverless functions using AWS Lambda (Python)

       Work with data lake architectures on AWS to support analytical use cases

       Ensure pipeline reliability, monitoring, and performance optimisation

 

Required Skills & Experience

       Strong hands-on experience with PySpark and AWS Glue

       Proven track record in ETL pipeline development and optimisation

       Experience orchestrating workflows with AWS Step Functions

       Proficiency in serverless development using AWS Lambda (Python)

       Good SQL skills and a solid understanding of data processing principles

 

Nice to Have

       Exposure to Java-based microservices

       Understanding of REST APIs and backend service integrations

       Experience working with AWS-based data lakes



Similar Jobs

16 Minutes Ago
Hybrid
Senior level
Senior level
Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
Technical lead responsible for designing, developing, testing, and maintaining enterprise backend services in C#/.NET. Lead API and integration development, drive CI/CD and testing practices, troubleshoot production issues, mentor developers, and collaborate with product and architecture teams in an Agile environment to deliver scalable, secure solutions.
Top Skills: .Net Core.Net FrameworkAzure DevopsC#Ci/CdGitGithub ActionsJenkinsRest ApisSQLWeb Services
Yesterday
In-Office
Internship
Internship
Artificial Intelligence • Big Data • Cloud • Information Technology • Software • Cybersecurity • Data Privacy
Winter software engineering intern responsible for identifying software bottlenecks and designing, developing, testing, deploying, maintaining, and improving software. The role includes managing project priorities, deadlines, and deliverables while working on Rubrik’s Continuous Product Development team in Pune. Candidates must meet academic eligibility requirements and be available for an onsite internship from January through May 2027.
Top Skills: CC++JavaPythonScala
Yesterday
In-Office
Entry level
Entry level
Cloud • Information Technology • Internet of Things • Machine Learning • Software • Cybersecurity • Infrastructure as a Service (IaaS)
Analyze customer technology strategies, business environments, and non-functional requirements to identify improvement opportunities. Collaborate with the Core 3 team to design commercially viable technical solutions, develop value propositions and business strategies, and create detailed technical offerings and proposals. Present solutions to customers, support negotiations, contribute to project implementation, and provide technical coaching and knowledge sharing.
Top Skills: Communication ProtocolsCotsHardware ArchitectureInfrastructure Capacity AnalysisSystem InterfacesVirtual Environments

What you need to know about the Bengaluru Tech Scene

Dubbed the "Silicon Valley of India," Bengaluru has emerged as the nation's leading hub for information technology and a go-to destination for startups. Home to tech giants like ISRO, Infosys, Wipro and HAL, the city attracts and cultivates a rich pool of tech talent, supported by numerous educational and research institutions including the Indian Institute of Science, Bangalore Institute of Technology, and the International Institute of Information Technology.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account