Lead Data Engineer

HighLevel Individual Duties:
Data Engineering
o Error handling / tracking / logging
Data Analysis
o Get access / understand where issues are in PHI (healthcare specific)
o Look at diagrams to understand the connection of data
o SQL queries
o Assess why things are failing
Data Engineer MustHaves (Concepts & Tools):
Databricks Pyspark / Python
Ascend.io REALLY nice to have
AWS would also be great 40 jobs on glue that are being migrated to Databricks
Day to Day: work with tech team to understanding requirements

Advantages of pyspark for data transformation

Expect Data Analysis

Should have healthcare

Read Architectural diagrams

Want a strong Data Engineer dont need to be a tech lead

Can do triage evaluate problems think through issues

Design data pipelines
Error handling/tracking
Data file exchange and production
Integration work with UHG
DataLakes
Databricks
Pyspark is more important than Databricks
Python more important that Databricks too
Ascend structure is easy to pick up
Write it in IDE check it in and promote it for review in system
SQL is needed for doing spark SQL scripts
Data pipeline tool Ascend
DevOps tools
Cloud exp. AWS is preferred
Glue is a
Healthcare is not a must but a huge plus