Exp: 5 to 12 Years
Job Description:
Please consider Python as ETL tool to read data from source perform all datatype & data transformation, error handling ,logging within python and load to data bricks managed delta tables.
Anyone we are hiring should be very good at python along with very good knowledge in Databricks & PySpark so that they can evaluate and perform transformation within Databricks as well. Candidate should be able to do almost everything in python including class ,objects, methods, dictionaries ,loops, Wheel files ,packages, should be able to connect to any source data including any DBs, streaming data.
Expertise in Azure services such as Azure SQL Database, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake
• The ideal candidate has extensive experience in data engineering, working with large-scale data infrastructures, and optimizing ETL processes
• Expertise in Azure services such as Azure SQL Database, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake
• At least 6+ years of experience in data engineering, with a focus on building data pipelines
• Proven experience with Spark, Scala, and other related technologies
• Familiarity with Could Warehouses, data lakes, storage formats.
• Strong knowledge of data modeling and ETL processes
• Experience leading data engineering projects.
• Proven experience with Databricks and Apache Spark
• Strong proficiency in Python
• Experience in Unix shell scripting and scheduling tools
