Senior Data Engineer Databricks Python Azure Data Factory CICD pipeline

Role: Senior Data Engineer (Databricks Python Azure Data Factory CI/CD pipeline)

Location: Remote

Job Type: C2C/1099 (Immediate requirement)

Responsibilities:

Design develop and maintain data pipelines using DataBricks and PySpark to process and manipulate large scale datasets.

Azure Data Factory

CI/CD Pipelines

Proven experience in optimizing Apache Spark batch processing workflows.

Extensive experience in building and maintaining streaming data pipelines.

Optimize and finetune existing DataBricks jobs and PySpark scripts for enhanced performance and reliability.

Troubleshoot issues related to data pipelines identify bottlenecks and implement effective solutions.

Implement best practices for data governance security and compliance within DataBricks environments.

Stay updated with industry trends and advancements in DataBricks and PySpark technologies to propose and implement innovative solutions.

Demonstrated expertise in optimizing systems for lowlatency and highthroughput performance.

Experience with using programming languages such as Python or Scala to implement advanced filtering logic in Databricks notebooks or scripts.

Familiarity with the principles of distributed systems and their application in message broking.

Collaborate with cross functional teams to gather requirements understand data needs and implement scalable solutions.

Requirements:

Bachelors or masters degree in computer science Engineering or a related field.

5 to 10 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.

Proficiency in PySpark and extensive handson experience in building and optimizing data pipelines using DataBricks.

Solid understanding of different components within DataBricks such as clusters notebooks jobs and libraries.