Data Engineer Lead - 3723

Data Engineer Lead

US (Remote)

JD-

Experience of End-to-End data platform implementation on Azure platform

Understand business processes and how they are modelled in various systems

Extensive development experience on Azure Synapse analytics, Azure Data Factory, Data Bricks and Spark

Must have good Data analysis skills (MS SQL Server)

Work experience on advanced programming languages Scala, Python and Spark

Database development skills for Azure Synapse

Experience and expertise on Design Document build and Architecture diagram build

Experience of doing impact analysis on existing data source systems

Experience and expertise in designing and implementing modular pipelines and reusable/configurable frameworks

Work with Data Engineering Lead, technology teams to understand requirements to create data pipelines

Implement data structures, workflows, and integrations between enterprise platforms to ensure the accurate and timely execution of business processes.

Develop and maintain scalable data pipelines and build out new API integrations to support continuing increases in data volume and complexity.

Managing and improving the performance of database, queries, tools, and solutions

Creating and maintaining data warehouse, databases, tables, SQL queries, and ingestion pipelines to power BI reports

Writing complex and efficient queries to transform raw data sources into easily accessible models for our teams and reporting platforms

Prepare data for predictive and prescriptive modelling

Identify and analyse data patterns

Identify ways to improve data reliability, efficiency, and quality

Work with analytics, data science, and wider engineering teams to help with automating data analysis and visualization needs, advise on transformation processes to populate data models, and explore ways to design and develop data infrastructure

Strong experience in data curation/aggregation/preparation using Azure Data Lake, Azure Data Factory and distilling data into meaningful insights with hands-on expertise on Azure and Snowflake/Databricks.

Create and maintain ETL specifications and process documentation to produce the required data deliverables data profiling source to target maps ETL flows

Performance tuning of ETL workflows and SQL scripts

Strong experience in Azure based EDW projects

Experience with Agile development methodologies

Good Verbal and Written communication skills Mandatory

Azure Synapse analytics, Azure Data Factory, Data Bricks and Spark

MS SQL Server

Scala, Python and Spark

Work experience preferably on Azure Cloud