Data Engineer

Responsibilities:

Data Architecture Design: Design and develop scalable, efficient, and robust data architecture solutions to support data storage, processing, and retrieval requirements.

Data Pipeline Development: Build and maintain data pipelines to efficiently extract, transform, and load (ETL) data from various sources into the data warehouse or data lake.

Data Integration: Integrate data from multiple systems, databases, and APIs, ensuring data consistency, accuracy, and quality.

Data Transformation and Cleansing: Perform data transformation, cleansing, and validation to ensure data integrity and conformity with defined business rules and standards.

Data Modeling: Design and implement data models that meet the analytical and reporting needs of the organization, including dimensional modeling, star schemas, and data marts.

Data Warehousing: Design and manage data warehousing solutions, including schema design, indexing, partitioning, and performance optimization.

Data Governance and Security: Implement data governance practices and ensure data security and privacy standards are upheld throughout the data lifecycle.

Performance Optimization: Identify and optimize performance bottlenecks in data processing and storage, including query optimization, indexing, and data partitioning.

Data Monitoring and Maintenance: Monitor data pipelines, data quality, and system performance, and proactively address any issues or anomalies.

Documentation: Create and maintain technical documentation, including data flow diagrams, system architecture, and data dictionaries.

Requirements:

Education and Experience:

Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.

Proven work experience as a data engineer or in a similar role.

Strong understanding of data engineering principles, techniques, and best practices.

Technical Skills:

Proficiency in programming languages like Python, Java, or Scala.

Experience with SQL and database technologies (e.g., Oracle, MySQL, PostgreSQL).

Knowledge of big data technologies and frameworks (e.g., Hadoop, Spark, Kafka).

Familiarity with cloud platforms and services (e.g., AWS, Azure, Google Cloud).

Understanding of data modeling techniques and concepts.

Experience with data integration tools and ETL frameworks (e.g., Apache Airflow, Informatica, Talend).

Knowledge of data warehousing concepts and methodologies.

Strong problem-solving and analytical skills.

Soft Skills:

Excellent communication and collaboration skills.

Ability to work effectively in a team environment.

Strong attention to detail and organizational skills.

Ability to manage multiple tasks and prioritize effectively.

Continuous learning mindset to keep up with evolving data engineering technologies and practices.