Lead Databricks Data Engineer

About Persistent

We are an AI-led platform-driven Digital Engineering and Enterprise Modernization partner combining deep technical expertise and industry experience to help our clients anticipate whats next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many industry-leading organizations across the world including 20 Fortune 50 companies and 4 of the 5 top banks in both the US and India and numerous innovators across the healthcare ecosystem.

Our disruptors mindset commitment to client success and agility to thrive in the dynamic environment have enabled us to sustain our growth momentum. Persistent has been recognized across top industry platforms for innovation leadership and inclusion. We reported $1654.4M FY26 revenue with 17.4% Y-o-Y growth. We have delivered 24sequential quarters of growth with $436.0M in Q2 FY27 revenue up 3.2% Q-o-Q and 16.2% Y-o-Y growth. Our 27500global team members located in 18 countries have been instrumental in helping the market leaders transform their industries. We have been recognized as theFastest Growing IT Services Brand Globally in the 2026 Brand Finance IT Services 25 Report. We named a Leader in theEverest Group Private Equity (PE) Services PEAK MatrixAssessment 2026andSoftware Product Engineering PEAK MatrixAssessment 2026.

About Position:

We are looking for a skilled Lead Databricks Data Engineer with strong expertise in Data Engineering Big Data technologies and the Databricks Lakehouse Platform. The ideal candidate will be responsible for designing developing and supporting scalable data pipelines enterprise analytics solutions and cloud-native data platforms. The role requires deep hands-on experience in Databricks PySpark SQL ETL/ELT frameworks and cloud technologies. You will work closely with architects data scientists business stakeholders and engineering teams to deliver high-performance scalable and reliable data solutions that enable enterprise-wide analytics and data-driven decision-making.

• Role: Lead Databricks Data Engineer
• Location: Bengaluru Pune
• Experience: 8 to 12 Years
• Job Type: Full Time Employment

What Youll Do:

• Design develop and maintain scalable ETL/ELT pipelines using Databricks.
• Build and optimize large-scale data processing workflows using PySpark and Spark SQL.
• Develop data ingestion frameworks for structured semi-structured and unstructured data sources.
• Implement batch and real-time data processing solutions on the Databricks Lakehouse Platform.
• Design and support Delta Lake implementations and Lakehouse architectures.
• Optimize Databricks workloads for performance scalability reliability and cost efficiency.
• Develop data transformation validation cleansing and reconciliation frameworks.
• Support enterprise data modeling and data warehousing initiatives.
• Implement monitoring alerting and troubleshooting mechanisms for data pipelines.
• Integrate Databricks with cloud services APIs enterprise applications and third-party systems.
• Develop cloud-native data solutions across Azure AWS or Google Cloud environments.
• Support CI/CD implementation and deployment automation for data engineering workloads.
• Collaborate with Architects Business Analysts Data Scientists and Product Owners to deliver business-driven data solutions.
• Participate in solution design discussions architecture reviews and code review processes.
• Support production deployments incident resolution and performance tuning activities.
• Create technical documentation design specifications and operational runbooks.
• Mentor junior engineers and promote data engineering best practices across teams.

Expertise Youll Bring:

• 812 years of experience in Data Engineering Big Data and Analytics platform development.
• Strong hands-on expertise with Databricks Lakehouse Platform and enterprise-scale data engineering solutions.
• Advanced proficiency in PySpark Spark SQL Python and SQL.
• Extensive experience designing and developing scalable ETL/ELT pipelines and data integration frameworks.
• Deep understanding of Delta Lake architecture transaction management and performance optimization techniques.
• Strong knowledge of Data Warehousing concepts dimensional modeling and enterprise analytics platforms.
• Experience working with structured semi-structured and unstructured data at scale.
• Hands-on experience building batch and real-time data processing solutions.
• Strong expertise in data ingestion transformation validation cleansing and reconciliation processes.
• Experience integrating enterprise applications APIs cloud services and diverse data sources.
• Familiarity with Azure Data Factory AWS Glue or equivalent cloud-native data integration services.
• Experience working with Azure AWS or Google Cloud Platform environments.
• Strong understanding of Git source control practices and collaborative development methodologies.
• Experience implementing and managing CI/CD pipelines for data engineering workloads.
• Knowledge of Kafka Event Hub or similar streaming technologies for real-time data processing.
• Familiarity with Unity Catalog metadata management and enterprise data governance frameworks.
• Strong understanding of schema design data modeling and query optimization techniques.
• Experience using Azure DevOps and deployment automation frameworks.
• Exposure to AI/ML data pipelines feature engineering and model data preparation workflows.
• Strong troubleshooting production support and root cause analysis capabilities.
• Experience optimizing Databricks workloads for enterprise-scale performance and reliability.
• Ability to independently lead medium-to-large data engineering initiatives.
• Experience mentoring team members and conducting technical reviews.
• Excellent analytical communication and stakeholder management skills.
• Strong commitment to code quality testing engineering best practices and continuous improvement.
• Ability to contribute to architecture discussions and strategic data platform initiatives.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties flexible work hours and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life personal accident and Mediclaim hospitalization for self spouse two children and parents

Values-Driven People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals including those with disabilities and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements please inform us during the application process or at any time during your employment

Lets unleash your full potential at Persistent- is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.