Lead Data Engineer

Job Title: Lead Data Engineer

Location: Scottsdale AZ (100% onsite)

Hire Type: Contract/FTE/CTH

VISA: No OPT CPT H1B

Musthave skill set:

• Spark S3 Glue AWS Redshift Python and StreamSets experience

Description:

We are seeking a highly skilled Lead Data Engineer to join our team in Scottsdale AZ. The ideal candidate will have extensive experience in enterprise data architecture and management with proficiency in Spark S3 Glue AWS Redshift Python and StreamSets.

Responsibilities:

• Lead and architect enterprisewide initiatives including system integration data migration transformation data warehouse build data mart build and data lakes implementation/support.
• Conceptualize design and implement streaming data pipelines using Spark Structured Streaming Delta Lake and Databricks.
• Develop and maintain data models including conceptual logical and physical data modeling with expertise in relational and dimensional data modeling.
• Troubleshoot and debug complex technical issues related to data pipelines and infrastructure.
• Collaborate with crossfunctional teams to ensure the successful deployment and optimization of big data applications in a highperformance enterprise environment.
• Utilize advanced SQL skills for data manipulation performance optimization and RDBMS schema design.
• Implement and maintain ETL/ELT processes using tools such as Data Migration Service.
• Work with AWS environment Lambda S3 and other cloud services for data processing and storage.
• Familiarity with data quality/validation frameworks such as Great Expectations is a plus.
• Experience with Gitlabs CloudWatch CI/CD pipelines and Lambda configuration and optimization.
• Communicate effectively with technology partners stakeholders team members and senior management to articulate technology solutions and constraints.

Qualifications:

• Bachelors degree in computer science Information Technology or related field.
• 68 years of IT experience focusing on enterprise data architecture and management.
• Proficiency in Spark S3 Glue AWS Redshift Python and StreamSets.
• Experience with Databricks Structured Streaming Delta Lake and Delta Live Tables.
• Advanced understanding of streaming data pipelines and ETL/ELT processes.
• Strong SQL skills with experience in RDBMS schema design and optimization.
• Experience working in a teambased environment and ability to thrive in a fastpaced dynamic organization.

If you possess the required skills and experience and are eager to join a dynamic team at Persistent Systems we encourage you to apply for this exciting opportunity.