Data Engineer (Azure, Datawarehouse)

Summary

Our client is looking for an experienced Senior Data Engineer to enhance their DataOps practice. This role will provide technical leadership as a senior team member of the company. You will be responsible for driving Azure based data processes with best practices. You will ensure that projects and products deliver cost effective, sustainable data solutions that align with the company's strategic direction. You will be expected to develop data models, perform code reviews, and develop other artifacts (pipelines) plus influence design decisions throughout the DataOps development lifecycle. A person that is "Hands On" with "Last Touch" experience is needed.

This is position is based in our Houston, Texas.

Requirements

At least 5 years of professional experience in the IT field and at least 3 years in a senior IT leadership role

Strategic thinker, able to develop and articulate a vision and direction in complex organizational and (IT) technical situations

Hands-on experience with data ingestion using Synapse ADF or Databricks

Experience with building successful pipelines with high reliability

Strong data modelling, design, warehousing, and business intelligence knowledge.

Experience with Delta Lake ingestion patterns (delta, parquet) within Medallion stages. Can architect solutions with SCD-2/Time Travel as needed.

Strong, current technical proficiency with applications architecture, frameworks, and enterprise solutions delivery. Awareness of new technological solutions and their application to PAAS.

Strong collaboration and leadership skills with the ability to coach and develop teams to meet new challenges

Mastery of core Data Engineering skills including - Data Analysis, Data Profiling, Data Modelling (ERD, Dimensional), Validation, Log Analysis, Data Lifecycle management, Master Data Management & implementation patterns, Data Integration patterns and methods, Streaming

Azure Synapse Partition Management and Loading

Mastery of technologies involved in data management is required (database management systems, data services, data lakes, etc.)

Experience of working with Agile Delivery teams and a knowledge of DevOps constructs

Affinity with the other Enterprise Architecture domains (process, application, infrastructure, security and information risk)

Delivery-oriented with focus on results. Must be a doer with a hands-on attitude. Enjoys seeing an automated and validated model in Production and understands the scope of effort for delivery

Organized and methodical. Ability to initiate, plan and develop according to plan while working with a degree of uncertainty and risk

Competencies

Demonstrable experience designing and implementing modern data warehouse/data lake solutions with an understanding of best practices

Previous work experience in a cloud-based environment (Azure preferred)

Advanced proficiency in writing complex SQL statements with manipulation of large structured and semi-structured datasets in a scalable, cost-effective manner

Proficiency in Python or other programmatic languages is preferred. Data Wrangling with code.

Creativity and problem-solving skills

Experience in ADF-Synapse pipeline orchestration (triggers) and monitoring (alerts/failed jobs)

Excellent and effective communication, customer service and time-management skills.

Ability to communicate complex information with understanding to a variety of audiences who have varying levels of experience and knowledge

Demonstrated success in delivering complex projects on time, on budget, and effectively using Scrum-Agile-DevOps-DataOps practices.

Experienced with DevOps deployments (pipelines, artifacts, batch scripting, JSON variables/JSON ARM templates) via CI/CD, Infrastructure as Code, YAML, Ansible or Terraform

Has done PR reviews for other Team members and has logically clarified junior team members' expected outcomes from code and SQL queries

Has a passion for collaboratively transforming business performance in project delivery through people, process, and technology, with a track record of results

Ability to maintain composure and adapt in a dynamic, fast-paced work environment, characterized by rapid change, minimal lead times, and multiple competing priorities.

Ability to successfully communicate and influence at all levels in an organization

Experienced with administrating ACL, Azure RBAC, RLS, OAuth2, MFA, Azure AD Groups, Service Principals, some Power BI Administration

Experienced with Azure Key Vault (rotating certificates and AKV security keys/secrets)

Education

College degree required with a BS in Computer Science, Information Technology, Computer Engineering or Mathematics highly desired.

Other Skills and Abilities

Innovative, high-energy leader with a passion to identify/develop leading edge technologies that have an immediate positive impact to business outcomes

Exceptional communication, leadership and influencing skills; ability to work independently in highly ambiguous environments, with both technical and business groups internally and externally