Databricks Specialist
Washington, DC - USA
Job Summary
Job Title: Databricks Specialist
Location: Washinton DC
Summary
The Senior Data Engineer / Databricks Specialist will support a federal project by developing designing and optimizing high-performance Spark applications and enterprise data infrastructure in AWS Databricks. The role focuses on scalable ETL/ELT pipelines using Python PySpark SQL and Delta Lake; Medallion Architecture; and secure reliable data solutions for mission-critical environments.
Responsibilities
- Develop and optimize scalable Spark applications in AWS Databricks using Python PySpark and SQL.
- Design build and maintain ETL/ELT pipelines using Medallion Architecture (Bronze Silver and Gold layers) ensuring data integrity completeness and lineage.
- Administer Databricks workspaces clusters scheduled jobs access controls Unity Catalog and notebooks.
- Manage AWS services including S3 Lambda IAM Aurora and Glue; provision infrastructure with Terraform.
- Optimize queries and Spark jobs; troubleshoot and debug large-scale data processing solutions.
- Integrate structured semi-structured and unstructured data from APIs and databases.
- Integrate security and compliance controls into DevSecOps CI/CD pipelines using Jenkins GitLab CI or equivalent tools supporting FISMA/FedRAMP requirements ATO reassessments and POA&M resolution.
- Implement data quality checks anomaly detection validation rules and automated lineage tracking.
- Collaborate with cross-functional Agile/Scrum teams in multi-contractor federal environments; provide technical leadership and production on-call support.
Required Qualifications
- At least 3 years of dedicated hands-on Databricks development experience.
- Production experience with PySpark Apache Spark Spark SQL and Delta Lake.
- Strong proficiency in Python and SQL and working knowledge of REST API-based data ingestion.
- Strong hands-on experience with AWS S3 IAM Lambda Glue and Aurora.
- Production experience with Terraform infrastructure as code and CI/CD tools such as Jenkins GitLab CI or equivalent.
- Experience working in Agile/Scrum delivery environments.
- Bachelors degree in Computer Science Information Technology Data Science or a related engineering discipline.
- Active or previous High-Risk Public Trust (Tier 4) clearance.
Preferred Qualifications
- Hands-on experience with Unity Catalog Delta Live Tables (DLT) and MLflow or Model Registry for AI/ML pipelines.
- Experience supporting federal government contracts and FISMA/FedRAMP environments.
Preferred Certifications
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
- Databricks Certified Machine Learning Associate
- AWS Certified Solutions Architect Associate or Professional
- AWS Certified Data Analytics Specialty
Required Experience:
IC
About Company
Red Cedar Consultancy, LLC We provide IT Services to Federal Customers such as Dept. of Homeland Security (DHS) United States Citizenship and Immigration Services (USCIS), Transportation Security Authority (TSA) and U.S Coast Guard (USCG), U.S Small Business Administration, U.S Treasu ... View more