Data Engineer III
Data Engineer III
Posted on :
16-10-2024
Employer Active
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
Client: Utility & Energy
Role: AWS Data Engineer IV
Location: Charlotte NC
Duration: 2436 Months
Core Technical Skills:
- 5 years of AWS experience
- AWS services S3 EMR Glue Jobs Lambda Athena CloudTrail SNS SQS CloudWatch Step Functions QuickSight
- Experience with Kafka/Messaging preferably Confluent Kafka
- Experience with EMR databases such as Glue Catalog Lake Formation Redshift DynamoDB and Aurora
- Experience with AWS data warehousing tools such as Amazon Redshift and Amazon Athena
- Proven track record in the design and implementation of data warehouse solutions using AWS
- Skilled in data modeling and executing ETL processes tailored for data warehousing
- Competence in developing and refining data pipelines within AWS
- Proficient in handling both realtime and batch data processing tasks
- Extensive understanding of database management fundamentals
- Expertise in creating alerts and automated solutions for handling production problems
- Tools and Languages Python Spark PySpark and Pandas
- Infrastructure as Code technology Terraform/CloudFormation
- Experience with Secrets Management Platform like Vault and AWS Secrets manager
- Experience with Event Driven Architecture
- DevOps pipeline (CI/CD); Bitbucket; Concourse
- Experience with RDBMS platforms and Strong proficiency with SQL
- Experience with Rest APIs and API gateway
- Deep knowledge of IAM roles and Policies
- Experience using AWS monitoring services like CloudWatch CloudTrail ad CloudWatch events
- Deep understanding of networking DNS TCP/IP and VPN
- Experience with AWS workflow orchestration tool like Airflow or Step Functions
AWS Data Engineer IV Additional Technical Skills (nice to have but not required for the role)
- Experience with native AWS technologies for data and analytics such as Kinesis OpenSearch
- Databases Document DB Mongo DB
- Hadoop platform (Hive; HBase; Druid)
- Java Scala Node JS
- Workflow Automation
- Experience transitioning on premise big data platforms into cloudbased platforms such as AWS
- Strong Background in Kubernetes Distributed Systems Microservice architecture and containers
Core Responsibilities
- Provides technical direction guides the team on key technical aspects and responsible for product tech delivery
- Lead the Design Build Test and Deployment of components
- Where applicable in collaboration with Lead Developers (Data Engineer Software Engineer Data Scientist Technical Test Lead)
- Understand requirements / use case to outline technical scope and lead delivery of technical solution
- Confirm required developers and skillsets specific to product
- Provides leadership direction peer review and accountability to developers on the product (key responsibility)
- Works closely with the Product Owner to align on delivery goals and timing
- Assists Product Owner with prioritizing and managing team backlog
- Collaborates with Data and Solution architects on key technical decisions
- The architecture and design to deliver the requirements and functionality
- Skilled in developing data pipelines focusing on longterm reliability and maintaining high data quality
- Designs data warehousing solutions with the enduser in mind ensuring ease of use without compromising on performance
- Manage and resolve issues in production data warehouse environments on AWS
Core Experience and Abilities
- Ability to perform hands on development and peer review for certain components / tech stack on the product
- Standing up of development instances and migration path (with required security access/roles)
- Develop components and related processes (e.g. data pipelines and associated ETL processes workflows)
- Lead implementation of integrated data quality framework
- Ensures optimal framework design and load testing scope to optimize performance (specifically for Big Data)
- Supports data scientist with test and validation of models
- Performs impact analysis and identifies risk to design changes
- Ability to build new data pipelines identify existing data gaps and provide automated solutions to deliver analytical capabilities and enriched data to applications
- Ability to implement data pipelines with the right attentiveness to durability and data quality
- Implements data warehousing products thinking of the end users experience (ease of use with right performance)
- Ensures Test Driven development
- 5 years of Experience leading teams to deliver complex products
- Strong technical skills and communication skills
- Strong skills with business stakeholder interactions
- Strong solutioning and architecture skills
- 5 years of Experience building real time data ingestion streams (event driven)
- Ensure data security and permissions solutions including data encryption user access controls and logging
Employment Type
Full Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
