Software Developer ETL 8079-3013
Software Developer ETL 8079-3013
Posted on :
26-10-2024
Employer Active
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
HM Note: This onsite role is three (3) days in office at managers discretion.
Scope
The Office of the Public Guardian and Trustee (OPGT) requires a Senior Software Developer ETL to perform data migration activities from the existing OPGT legacy applications to the new Dynamics 365 OPGT solution and data warehouse/lakehouse.
and nbsp;
Assignment Deliverables
As a member of the data migration team you will be responsible to migrate the data from the existing OPGT legacy applications to the new Dynamics 365 OPGT solution and the new data warehouse/lakehouse. A highlevel list of deliverables for the data migration team follows:
Data Analysis: analyze the existing data in the legacy applications understand its structure quality and relationships and help in designing an appropriate migration strategy;
- Data Mapping and Transformation: map the data elements from the legacy application to the corresponding entities and fields in Dynamics 365 CE F and amp;O and Data Lakehouse. Handle necessary data transformations ensuring compatibility and consistency between the legacy data and the target system;
- Data Extraction: help extract the required data from the legacy application develop and implement extraction processes to retrieve data from various sources such as databases files APIs or other relevant legacy systems;
- Data Cleansing and Validation: cleanse and validate the extracted data to ensure its accuracy completeness and consistency. Help with identifying and resolving data quality issues performing deduplication and applying business rules to ensure the integrity of the migrated data;
- Data Migration Strategy and Execution: review the present migration strategy that outlines the overall approach sequence and timeline for migrating the data from the legacy application to Dynamics 365 using a deltaload approach; execute the migration plan efficiently managing data transfers and ensuring minimal disruption to ongoing operations;
- Data Testing and Quality Assurance: conduct thorough testing to verify the accuracy and integrity of the migrated data; define test cases perform data reconciliation and address any issues or discrepancies that arise during the testing phase; develop KPIs to report on the progress completeness and quality of the data migration effort;
- Documentation: document the entire data migration process including data mapping rules transformation logic migration scripts and any specific configurations;
- Ongoing Support: provide postmigration support analyze and address datarelated issues or questions; help optimize data management processes in the new environment;
- Data Lakehouse: help expand the current data lakehouse implementation and help build a gold layer for reporting; conduct data analysis and profiling; develop data transformation and cleansing pipelines; implement data integration and harmonization;
- Other duties as assigned;
The Vendors Personnel will also be required to:
Complete work and achieve milestones within the assigned deadlines;
- Notify the Cluster/Ministry project Manager in writing of any issues or other material concerns related to the Assignment Deliverables as soon as he/she becomes aware of them
- Submit Deliverables for the Cluster/Ministry approval as they are completed.
- Comply with the Ontario Government and the Cluster/Ministry security procedures and practices
- Comply with the Ontario Government and the Cluster/Ministry architecture/technology standards and best practices
- Comply with the Ontario Government and the Cluster/Ministry Conflict of Interest and Confidentiality Guidelines
- Provide knowledge and skill transfer to a designated Cluster/Ministry staff; and
- Comply with the Ontario Government I and amp;IT Directive Operational Policy on the I and amp;IT Project Gateway Process and other applicable Guidelines Standards and Procedures.
Skills
Experience and Skill Set Requirements
Experience and Skillset Requirements
and nbsp;
Mandatory Requirements
5 years of proven working experience in an ETL role; strong understanding of ETL principles including data extraction transformation and loading processes; knowledge of common ETL design patterns. Understanding of data pipeline architectures Azure workflow orchestration tools and concepts related to data ingestion transformation and movement.
- Proficiency in Azure Data Factory Azure Synapse workspaces PolyBase including knowledge of pipeline creation data flows integration runtimes triggers and monitoring. and nbsp;
- Strong SQL skills and experience working with Azure SQL Databases and Dataverse; good understanding of Azure storage concepts and technologies.
- Proficiency in scripting languages like Python experience with Azurespecific scripting using PowerShell or Azure CLI.
- Expert proficiency with data manipulation languages (TSQL PL/SQL) data definition languages physical database design data modeling query performance analysis and amp; tuning
- Knowledge of integration technologies commonly used with Dynamics such as DataVerse / Common Data Service (CDS) Data Entities and APIs.
- Experience with continuous integration/continuous deployment (CI/CD) processes around DevOps data workflows Synapse workspaces.
- Familiarity with data warehousing concepts and experience working with Azure Synapse Analytics or similar platforms for building and managing largescale data warehousing/lakehouse solutions.
- Experience with SSIS SSRS PowerBI
NicetoHave Requirements
Azure cloud certifications (e.g. Azure fundamentals Azure Data Engineer associate Azure Database Administrator associate)
- Experience with PowerApps platform Power Automate Dynamics CE and F and amp;O
Desired Skills and Experience
5 years of proven working experience in an ETL role; strong understanding of ETL principles including data extraction transformation and loading processes; knowledge of common ETL design patterns. Understanding of data pipeline architectures Azure workflow orchestration tools and concepts related to data ingestion transformation and movement.
- Experience in integrating various data sources and systems both onpremises and in the cloud using Azure ETL services or other ETL tools
- Knowledge of integration technologies commonly used with Dynamics such as DataVerse / Common Data Service (CDS) Data Entities and APIs.
- Expertise in data transformation techniques such as data cleansing aggregation enrichment and normalization using Azure cloud technologies
- Understanding of data quality management practices including data profiling data validation and error handling within ETL processes. and nbsp;
- Understanding of data governance principles data privacy regulations and experience working with highsensitivity data and knowledge of best practices for data security and compliance in Azure. and nbsp;
- Ability to monitor and troubleshoot ETL processes optimize query performance and implement efficient data processing techniques in Azure.
- Proficiency in Azure Data Factory Azure Synapse workspaces PolyBase including knowledge of pipeline creation data flows integration runtimes triggers and monitoring. and nbsp;
- Strong SQL skills and experience working with Azure SQL Databases and Dataverse; good understanding of Azure storage concepts and technologies.
- Proficiency in scripting languages like Python and experience with Azurespecific scripting using PowerShell or Azure CLI. and nbsp;
- Expert in data manipulation languages (TSQL PL/SQL) data definition languages physical database design data modelling query performance analysis and amp; tuning
- Familiarity with version control systems (e.g. Azure Repos) and collaboration tools (e.g. Azure DevOps) for managing code tracking changes and collaborating with team members.
- Experience with continuous integration/continuous deployment (CI/CD) processes around DevOps data workflows Synapse workspaces.
- Experience with SQL Server Management Studio Azure data management tools XRM toolbox data modeling tools (preferably ERWIN). and nbsp;
- Familiarity with data warehousing concepts and experience working with Azure Synapse Analytics or similar platforms for building and managing largescale data warehousing/lakehouse solutions.
- Experience with SSIS SSRS PowerBI
and nbsp;
Resumes Evaluation/Criteria:
Criteria 1: Data Migration ETL 40 Points
Demonstrated experience with ETL development data pipelines workflow orchestration and data ingestion transformation and movement
- Demonstrated experience in integrating various data sources and systems both onpremises and in the cloud using Azure ETL services or other ETL tools
- Demonstrated experience working with Azure Data Factory Azure Synapse workspaces PolyBase including knowledge of pipeline creation data flows integration runtimes triggers and monitoring. and nbsp;
- Demonstrated experience with data manipulation languages (TSQL PL/SQL) data definition languages query performance analysis and amp; tuning
- Demonstrated experience with SQL Server Oracle Azure SQL Databases
- Demonstrated experience with data modeling tools (preferably ERWIN)
- Demonstrated experience in scripting languages like Python and with Azurespecific scripting using PowerShell or Azure CLI. and nbsp;
- Experience with software development lifecycle
- Experience with data modeling physical database design data flow diagrams
and nbsp;
Criteria 2: Data Warehouse and Reporting 20 Points
Demonstrated experience working with Azure Synapse Analytics or similar platforms for building and managing largescale data warehousing/lakehouse solutions
- Experience with data warehousing modelling concepts such as star and snowflake schemas
- Experience with SSIS SSRS PowerBI
- Experience with supporting a data warehouse in a production environment
Criteria 3: and nbsp;Azure Platform and nbsp; 20 Points
Experience with Azure Data Factory (ADF) and Synapse Workspaces
- Demonstrated experience Azure data management tools DevOps Synapse Studio
- Experience in Azure resource configuration and administration such as Azure Data Lake Blob Storage Key Vault Application Insight resources resource groups and subscriptions.
- Familiar with Azure cloud platform
- Azure cloud certifications
and nbsp;
Criteria 4: and nbsp;Dynamics 365 and nbsp; 10 Points
and nbsp;Demonstrated Demonstrated experience working with integration technologies commonly used with Dynamics such as DataVerse / Common Data Service (CDS) Data Entities and APIs.
- Demonstrated experience with PowerApps platform Power Automate Dynamics CE and amp;F and amp;O
Criteria 5: DevOps and CI/CD 10 Points
Demonstrated experience with continuous integration/continuous deployment (CI/CD) tools and processes around DevOps data workflows Synapse workspaces.
and nbsp;
Knowledge Transfer
and nbsp;
What needs to be KT
- Document the tasks executed and in process of execution as a member of the data migration team using the approach and tools required by the project manager.
and nbsp;
To whom
- Project Manager/Team Members
and nbsp;
When
- 1:1 meetings / team meetings / documentation in DevOps Wiki and boards throughout the duration of the project life cycle.
Employment Type
Full Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
