Cloud Engineer

Job Title: Cloud Engineer

Location: Minneapolis MN

Duration: 12months

Job Description;

1. ETL and Data Pipeline Development

Design develop and optimize scalable ETL processes using Python Apache Spark and Azure Synapse.

Build and manage Azure Data Factory pipelines to orchestrate complex data workflows.

Use SQL Pools and Spark Pools within Synapse to manage and process large datasets efficiently.

Implement Data Warehousing solutions using Azure Synapse ytics to provide structured and queryable data layers

Ensure the data platform supports realtime and batch AI/Client data requirements.

2. Azure Cloud Development & CI/CD Deployment

Build configure and manage CI/CD pipelines on Azure DevOps for ETL and data processing tasks.

Automate infrastructure provisioning testing and deployment using InfrastructureasCode (IaC) tools like ARM templates or Terraform.

Optimize Azure Data Lake Storage (ADLS Gen2) to store and manage raw and processed data efficiently ensuring proper access control and data security.

3. CrossFunctional Collaboration

Collaborate with Data Scientists Data Engineers Client Engineers and Business ysts to translate business requirements into data solutions.

Work with the DevOps and Security teams to ensure smooth and secure deployment of applications and pipelines.

Act as the technical lead in designing developing and implementing data solutions mentoring junior team members.

4. Data Engineering and API Development

Develop and integrate with external and internal APIs for data ingestion and data exchange.

Build test and deploy RESTful APIs for secure data access.

Use Kubernetes for containerizing and deploying data processing applications.

Manage data storage and transformation to support advanced Data Science and AI/Client models.

5. Agile Project Management

Participate in and lead Agile ceremonies such as sprint planning daily standups and retrospectives.

Collaborate with crossfunctional teams in iterative development to ensure highquality and timely feature delivery.

Adapt to changing project priorities and business needs in an Agile environment.

What ss/technologies are required (please include the number of years of experience required)

1. Technical Ss:

Expertise in Python and Apache Spark for largescale data processing.

Strong experience in Azure Synapse ytics including SQL Pools and Spark Pools.

Advanced proficiency in Azure Data Factory for ETL pipeline orchestration and management.

Knowledge of Data Warehousing principles with handson experience building solutions on Azure.

Experience with SQL including complex queries optimization and performance tuning.

Familiarity with CI/CD tools like Azure DevOps and managing infrastructure in Azure Cloud.

Experience in Java for API integration and microservices architecture.

Handson knowledge of Kubernetes for containerized data processing environments.

Proficiency in working with Azure Data Lake Storage (ADLS) Gen2 for data storage and management.

Experience working with APIs (REST SOAP) and building APIbased data integrations.

2. Agile and CrossFunctional Ss:

Experience working in an Agile environment using Scrum or Kanban.

Ability to lead mentor and coach junior developers in the team.

Strong collaboration ss to work with data scientists ysts and crossfunctional teams to deliver endtoend data solutions.

3. Behavioral Ss:

Strong ytical and problemsolving ss with a pion for datadriven solutions.

Excellent communication and presentation ss able to explain complex technical concepts to nontechnical stakeholders.

Ability to work in a fastpaced dynamic environment with changing priorities.

Selfmotivated and resultsoriented with attention to detail.

12 years of experience in software development with 5 years in a lead developer role.

Lead Developer with advanced ss in Python Apache Spark Azure Synapse and Azure Data Engineering services to develop and manage ETL pipelines

and data processing solutions that support AI/Client initiatives. The ideal candidate will have strong expertise in Azure Cloud including SQL Data Factory

SQL Pools Spark Pools and Data Warehousing. A background in CI/CD processes crossfunctional collaboration and odologies is essential. In

addition knowledge of Data Science Java Kubernetes Azure Data Lake Storage (ADLS) Gen2 and API development is required.

Prior experience in leading crossfunctional teams and delivering complex software solutions.

What ss/attributes are preferred (these are a desired not required)

Azure certifications in data engineering or cloud architecture.

Experience deploying AI/Client models on cloud platforms.

Familiarity with Data Governance best practices ensuring compliance with data privacy regulations.

Expertise in Python and Apache Spark for large-scale data processing. - Strong experience in Azure Synapse ytics, including SQL Pools and Spark Pools