Data Architect
Data Architect
Posted on :
06-12-2024
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
We are seeking an experienced Data Architect to design and implement enterprise data solutions using Microsoft Fabric and Azure Databricks for integration with statelevel systems. This role will focus on creating scalable data architecture that enables seamless data flow between IES Gateway and our analytics platform. The ideal candidate will have deep expertise in modern data architecture with specific experience in Microsofts data platform and Delta Lake architecture.
Key Responsibilities:
Data Architecture:
Design endtoend data architecture leveraging Microsoft Fabrics capabilities.
Design data flows within the Microsoft Fabric environment.
Implement OneLake storage strategies.
Configure Synapse Analytics workspaces.
Establish Power BI integration patterns.
Integration Design:
Architect data integration patterns between IES Gateway and the analytics platform using Azure Databricks and Microsoft Fabric.
Design Delta Lake architecture for IES Gateway data.
Implement medallion architecture (Bronze/Silver/Gold layers).
Create realtime data ingestion patterns.
Establish data quality frameworks.
Lakehouse Architecture:
Implement modern data lakehouse architecture using Delta Lake ensuring data reliability and performance.
Data Governance:
Establish data governance frameworks incorporating Microsoft Purview for data quality lineage and compliance.
Implement rowlevel security.
Configure Microsoft Purview policies.
Establish data masking for sensitive information.
Design audit logging mechanisms.
Pipeline Development:
Design scalable data pipelines using Azure Databricks for ETL/ELT processes and realtime data integration.
Performance Optimization:
Implement performance tuning strategies for largescale data processing and analytics workloads.
Optimize Spark configurations.
Implement partitioning strategies.
Design caching mechanisms.
Establish monitoring frameworks.
Security Framework:
Design and implement security patterns aligned with federal and state requirements for sensitive data handling.
Required Qualifications:
Education:
Bachelors degree in computer science or related field.
Experience:
6 years of experience in data architecture and engineering.
2 years handson experience with Azure Databricks and Spark.
Recent experience with Microsoft Fabric platform.
Technical Skills:
Microsoft Fabric Expertise:
Data Integration: Combining and cleansing data from various sources.
Data Pipeline Management: Creating orchestrating and troubleshooting data pipelines.
Analytics Reporting: Building and delivering detailed reports and dashboards to derive meaningful insights from large datasets.
Data Visualization Techniques: Representing data graphically in impactful and informative ways.
Optimization and Security: Optimizing queries improving performance and securing data
Azure Databricks Experience:
Apache Spark Proficiency: Utilizing Spark for largescale data processing and analytics.
Data Engineering: Building and managing data pipelines including ETL (Extract Transform Load) processes.
Delta Lake: Implementing Delta Lake for data versioning ACID transactions and schema enforcement.
Data Analysis and Visualization: Using Databricks notebooks for exploratory data analysis (EDA) and creating visualizations.
Cluster Management: Configuring and managing Databricks clusters for optimized performance. (Ex: autoscaling and automatic termination)
Integration with Azure Services: Integrating Databricks with other Azure services like Azure Data Lake Azure SQL Database and Azure Synapse Analytics.
Machine Learning: Developing and deploying machine learning models using Databricks MLflow and other tools.
Data Governance: Implementing data governance practices using Unity Catalog and Microsoft Purview
Programming & Query Languages:
SQL: Proficiency in SQL for querying and managing databases including skills in SELECT statements JOINs subqueries and window functions12.
Python: Using Python for data manipulation analysis and scripting including libraries like Pandas NumPy and PySpark
Data Modeling:
Dimensional modeling
Realtime data modeling patterns
Soft Skills:
Strong analytical and problemsolving abilities
Excellent communication skills for technical and nontechnical audiences
Experience working with government stakeholders
Preferred Experience:
Azure DevOps
Infrastructure as Code (Terraform)
CI/CD for data pipelines
Data mesh architecture
Employment Type
Full Time
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
Similar Jobs
Dr. Job is an online platform that connects employers with skilled job seekers, facilitating the search for job opportunities and top talent. Established in 2015. Dr. Job Pro has emerged as the world premier job portal, attracting thousands of job seekers every day from all over the world.
Follow Dr.Job
Dr Job FZ LLC. 2024 © All Rights Reserved