Data Engineering Lead - Data Bricks

This position will be engaged in design and deploy of largescale data processing frameworks and pipelines on Data bricks to support enterprisewide BI and data science initiatives. It will also provide user training and data coaching for effective Data bricks usage.

Locations

We are located at Austin (USA) Singapore Gurugram (India) Ahmedabad (India) and Hyderabad (India).

Job Location Hyderabad Indore Ahmadabad Pune Gurgaon

We are looking for you!

Are you passionate about leveraging data to drive business insights and innovation Do you thrive in a dynamic collaborative environment where you can contribute your expertise to building scalable data solutions Data Bricks is seeking a talented Data Engineer to join our team and play a pivotal role in designing and implementing data pipelines and infrastructure for our cloudbased analytics platform.

What you will do

• Create MLOps frameworks for model development training deployment and monitoring.
  

• Optimize Databricks clusters for performance scalability and resource efficiency.
  

• Monitor cluster health performance metrics and resource usage adjusting cluster sizes as needed.
  

• Design efficient data processing systems and pipelines using Data bricks APIs and other cloud services.
  

• Act as the Data bricks expert and uphold platform standards.
  

• Collaborate with Data and Platform architects on defining the product roadmap.
  

• Build and maintain scalable infrastructure on cloud platforms such as AWS Azure or Google Cloud to support data processing and analytics workloads.
  

• Implement data governance best practices to ensure data quality reliability and security throughout the data lifecycle.
  

• Monitor and troubleshoot data pipelines to identify and resolve performance bottlenecks data integrity issues and other operational challenges.
  

What you ll bring

• Bachelor s degree in computer science or related field.
  

• 7 years of IT experience including 5 years in Data bricks development and administration Spark and cloud platforms (preferably Azure).
  

• Proficiency in Databricks admin and development with expertise in Resource and Performance Management.
  

• Expertise in test automation continuous integration Docker GitHub Jenkins and Unix/Linux.
  

• Leadership in large project initiatives willingness to mentor and support for team members from different time zones.
  

You will have advantage if you bring

• Experience with Kafka and Spark and strong in Python and Pyspark.
  

• Handson experience in machine learning pipelines and scripting languages.
  

• Familiarity with Java Python SQL/NoSQL databases and Big Data tools.
  

• Passion for new tech continuous improvement and collaborative problemsolving.
  

• Proactive strong work ethic and team player.
  

1. Culture:

• Open Door Policy: Encourages open communication and accessibility to management.
  

• Open Office Floor Plan: Fosters a collaborative and interactive work environment.
  

• Flexible Working Hours: Allows employees to have flexibility in their work schedules.
  

• Employee Referral Bonus: Rewards employees for referring qualified candidates.
  

• Appraisal Process Twice a Year: Provides regular performance evaluations and feedback.
  

2. Inclusivity and Diversity:

• Hiring practices that promote diversity: Ensures a diverse and inclusive workforce.
  

• Mandatory POSH training: Promotes a safe and respectful work environment.
  

3. Health Insurance and Wellness Benefits:

• GMC and Term Insurance: Offers medical coverage and financial protection.
  

• Health Insurance: Provides coverage for medical expenses.
  

• Disability Insurance: Offers financial support in case of disability.