MS Azure with ADFDatabricks Senior SpecialistSenior Consultant
MS Azure with ADFDatabricks Senior SpecialistSenior Consultant
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
3-8years
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
We are an IT Solutions Integrator/Consulting Firm helping our clients hire the right professional for an exciting long term project. Here are a few details.
Requirements
We are seeking a skilled Azure Data Engineer to design develop and optimize our data solutions on Microsoft Azure focusing on Azure Data Factory (ADF) and Azure Databricks. This role requires an indepth understanding of Azure cloud services data transformation and pipeline development enabling the organization to leverage data for insights and decisionmaking.
Key Responsibilities
Data Pipeline Development and Management
- Design and implement scalable reliable and highperformance data pipelines using Azure Data Factory (ADF) to ingest transform and load data from various data sources.
- Develop endtoend ETL/ELT processes within ADF ensuring data quality accuracy and consistency.
- Design and implement scalable reliable and highperformance data pipelines using Azure Data Factory (ADF) to ingest transform and load data from various data sources.
Data Transformation and Processing with Databricks
- Utilize Azure Databricks for data wrangling cleaning and transformations to prepare data for analytics and machine learning.
- Implement Sparkbased data engineering workflows within Databricks optimizing performance and cost efficiency.
- Collaborate with data scientists to prepare data for advanced analytics and machine learning models.
- Utilize Azure Databricks for data wrangling cleaning and transformations to prepare data for analytics and machine learning.
Database and Data Storage Optimization
- Work with Azure Data Lake Storage (ADLS) SQL Databases and other Azure storage solutions to manage data and ensure efficient access.
- Design and implement optimized data storage solutions that support the scalability and agility of data processing workflows.
- Work with Azure Data Lake Storage (ADLS) SQL Databases and other Azure storage solutions to manage data and ensure efficient access.
Data Governance and Security
- Implement robust data governance practices ensuring compliance with data security privacy policies and data retention requirements.
- Apply access control mechanisms data masking and encryption strategies to protect sensitive data in ADF and Databricks.
- Implement robust data governance practices ensuring compliance with data security privacy policies and data retention requirements.
Performance Tuning and Optimization
- Monitor and improve the performance of ETL/ELT processes within ADF and Databricks.
- Troubleshoot and resolve issues related to data pipelines storage and processing ensuring high availability and reliability.
- Monitor and improve the performance of ETL/ELT processes within ADF and Databricks.
Collaboration and Documentation
- Collaborate with data architects analysts and business stakeholders to define data requirements and develop solutions that align with business goals.
- Document processes configurations and best practices to facilitate knowledge sharing within the team.
- Collaborate with data architects analysts and business stakeholders to define data requirements and develop solutions that align with business goals.
Required Qualifications
- Bachelor s degree in Computer Science Information Systems or related field.
- 3 years of experience in data engineering or similar role with a focus on Microsoft Azure.
- Proficiency in Azure Data Factory for ETL/ELT processes including activities like data ingestion transformation scheduling and monitoring.
- Handson experience with Azure Databricks for data processing transformation and pipeline development using Spark.
- Strong knowledge of SQL and Python/Scala for data manipulation scripting and automation within Databricks.
- Familiarity with Azure Data Lake Storage (ADLS) and other Azure storage solutions.
- Experience in performance tuning and cost optimization of data workflows on cloud infrastructure.
- Knowledge of Azure DevOps for CI/CD pipeline management version control and workflow automation.
Preferred Qualifications
- Azure Certifications (e.g. Microsoft Certified: Azure Data Engineer Azure Fundamentals).
- Experience in realtime data processing using Azure Stream Analytics or Event Hub.
- Familiarity with Power BI or other visualization tools for reporting and dashboarding.
- Understanding of data governance and compliance standards (GDPR HIPAA etc.).
Benefits
Bachelor s degree in Computer Science, Engineering, or a related field (or equivalent experience). 2+ years of hands-on experience in PySpark, Spark SQL, and Spark DataFrames. Proficiency in Python programming, with experience in data manipulation and processing. Strong knowledge of Apache Spark architecture and experience working with large datasets in a distributed environment. Experience with SQL and relational databases, including query optimization. Familiarity with ETL frameworks and data processing tools (e.g., Hadoop, Hive, Kafka). Experience in cloud platforms such as AWS, Azure, or Google Cloud and their respective big data services. Understanding of data lake and data warehouse concepts and best practices. Knowledge of data partitioning, caching, and other optimization techniques in Spark. Strong problem-solving and debugging skills, with attention to detail and accuracy.
Employment Type
Full Time
