Data Engineer

Roles and responsibilities

Responsible for the maintenance, improvement, cleaning, and manipulation of data in the business's operational and analytics databases.

Within the functional data integration team the Data-driven Engineer works in order to understand and aid in building the business' data collection systems.

Tasks

▪ Provide technical guidance related to data architecture, data models and meta data management

▪ Maintain architectures such as databases and large-scale processing systems

▪ Discover opportunities for data acquisition

▪ Develop data set processes for data modelling, mining and production

▪ Recommend ways to improve data reliability, efficiency and quality

▪ Evaluate datasets for accuracy and quality

▪ Develop best practices for data management, maintenance, reporting and security

▪ Develop and deploy of innovative big data platforms for advanced analytics and data processing

▪ Maintain optimized and highly available data pipelines that facilitate deeper analysis and reporting by the Data and Analytics department

Data Architecture Design

Design and implement scalable data architectures to support data storage, processing, and analytics.
Develop data models and schemas to efficiently organize and store data.
Data Integration

Build and maintain ETL (Extract, Transform, Load) processes to integrate data from various sources into data warehouses or lakes.
Collaborate with data architects and data scientists to understand data requirements and integrate new data sources.
Data Pipeline Development

Create and optimize data pipelines for real-time and batch processing to ensure timely data availability.
Use tools and frameworks like Apache Kafka, Apache Airflow, or similar for data orchestration and scheduling.
Database Management

Manage and maintain databases, ensuring data integrity, security, and accessibility.
Monitor database performance and implement tuning and optimization strategies.

Desired candidate profile

• Industry Knowledge: Familiarity with industries such as finance, healthcare, e-commerce, or technology.

Technical Skills

• Programming Languages: Proficiency in programming languages such as Python, Java, or Scala for data processing and transformation.
• Database Management: Experience with relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).
• ETL Tools: Familiarity with ETL tools (e.g., Apache NiFi, Talend, Informatica) for data integration processes.
• Big Data Technologies: Knowledge of big data frameworks like Apache Hadoop, Apache Spark, or cloud-based solutions (e.g., AWS, Azure, Google Cloud).
• Data Modeling: Experience in data modeling and database schema design.

Data Management Skills

• Data Quality: Strong understanding of data quality assurance practices and data governance principles.
• Performance Optimization: Skills in optimizing data pipelines for efficiency and scalability.

Communication and Collaboration

• Team Collaboration: Ability to work collaboratively with data analysts, data scientists, and business stakeholders to gather requirements and deliver solutions.
• Documentation: Strong written communication skills for documenting data processes and workflows.

Personal Attributes

• Analytical Thinker: Strong problem-solving and analytical skills to address complex data challenges.
• Detail-Oriented: High attention to detail to ensure accuracy and reliability in data engineering tasks.
• Adaptability: Willingness to learn new technologies and adapt to changing project requirements.

Responsibilities

• Design, build, and maintain scalable data pipelines and architectures.
• Collaborate with cross-functional teams to ensure data integration and accessibility.
• Monitor and optimize database performance and data workflows.
• Implement data quality checks and validation processes.