Lead Data Architect Data Engineer
Sacramento, CA - USA
Job Summary
We are seeking a hands-on Lead Data Architect/Data Engineer for a hybrid assignment with our Sacramento California-based client and are open to candidates within California
to help define architecture and deliver production-grade pipelines across multiple phases of a Databricks platform.
Responsibilities include:
* Build and maintain automated ingestion and transformation pipelines that bring data from:
o public portals and open data sources
o REST APIs
o internal departmental databases and systems
into Delta tables in Databricks.
* Partner with the Program Manager to establish and maintain:
o platform architecture
o naming conventions and standards
o technical documentation
* Collaborate with the Data Hub governance group to support decisions on:
o platform and data architecture
o data sharing patterns
o data contracts and data ownership
o data quality and reliability standards
* Design recommendations for protecting sensitive data including:
o row-level and column-level access controls
o separation of confidential/internal data from public datasets
* Improve reliability through:
o monitoring and alerting
o data quality checks and validation frameworks
* Enable discoverability and usability by publishing datasets and documentation that support plain-language exploration and AI-enabled analysis tools.
* Write and maintain skill files (instruction files for AI coding assistants) to enforce consistent pipeline patterns and development standards.
* Support and enable other departments to build pipelines using shared templates and best practices.
Requirements:
* Strong leadership experience including executive-level stakeholder engagement
* 7 years in data engineering data architecture or closely related roles
* Proven experience delivering large-scale enterprise data engineering / architecture initiatives
* Strong hands-on expertise in Python and SQL
* Experience with Apache Spark or cloud data platforms such as:
o Databricks (strongly preferred)
o Snowflake BigQuery or similar
* Experience integrating data from REST APIs databases and other source systems
* Excellent communication skills with strong writing and documentation practices
Preferred
* Hands-on Databricks experience with:
o Delta Lake
o Unity Catalog
o Databricks Jobs
* Experience governing sensitive/confidential data (access control masking privacy requirements)
* Experience operating within a data governance program (stewardship data contracts cross-agency sharing agreements)
* Experience writing skill files prompts or instruction sets for AI coding assistants
* Experience with GIS / location-based datasets
Required Experience:
Senior IC
About Company
Are you looking to hire? At eStaff LLC, we take all of the stress out of the search, with our experienced job recruiters able to staff all types of technical positions with highly skilled professionals. 24 hour on call.