Sr Data EngineerData Architect

Azure Services
Azure DataBricks
Python
Azure Data Factory
GitLab
Data Modeling
Data Analysis
Test Automation
SQL
Snowflake

UST is looking for a Sr. Data Engineer/Data Architect who is self driven can lead a team of Data engineers working on an OnshoreOffshore model collaborating with customer IT SMEs Business partners

Job responsibilities
Gather requirements working closely with IT SMEs Business stakeholders
Prepare technical design document coordinate with offshore team on development activities
Design build and deploy efficient & reliable data pipelines to move and transform data
Optimize existing pipelines and maintenance of all sourcerelated data pipelines
Support oncall shift as needed to support the team
Build and Deploy extensive data quality checks to ensure high quality of data
Build highperformance scalable data warehouses and data products
Partner with leadership engineers product managers and data scientists to understand data needs.
Improve existing data movement frameworks and add more features
Extreme focus on automation
Expert level SQL skills
Data Modeling
CI/CD Pipeline and DevOps practices
Agile fundamentals
Prior use of Jira and GitLab

Sharing the Questions to pass to the vendor for shortlisting
What are the different Normal Forms in Database
First Normal Form (1NF) 2nd Normal Form (2NF) Third Normal Form (3NF) &BoyceCodd Normal Form (BCNF)
2. Describe the Architecture of Snowflake
1. Compute Layer
2. Storage Layer
3. Cloud Services Layer
3. What are the key components of ADF
Pipelines Activities DataSets Linked services Data flows and Integration Runtimes
4. What is a Dataframe in Azure Databricks
Dataframe refers to a type of table used to store data within Databricks during runtime

Please screen the resume on the following sections:

2. What does scaling up and scaling out mean in Snowflake

4. What are the different types of activities in Azure Data Factory (ADF)

5. What is the difference between a dataset and a linked service in Azure Data Factory (ADF)

6. Explain the difference between a pipeline and an activity in Azure Data Factory.

7. What are global protected and private attributes in Python