GCP Data Engineer
Overview:
The GCP Data Engineer plays a crucial role in designing implementing and managing data processing systems leveraging Google Cloud Platform (GCP) services. This position is essential for ensuring efficient data management processing and analysis to support the organizations datadriven decisionmaking processes and solutions.
Key Responsibilities:
- Design develop and deploy GCPbased data processing systems and solutions.
- Implement scalable and reliable data pipelines for ingesting processing and storing large volumes of data.
- Optimize data storage and retrieval processes using GCP storage solutions.
- Collaborate with data analysts to understand data requirements and implement appropriate solutions.
- Ensure data integrity security and compliance with regulatory requirements.
- Monitor and troubleshoot data processing systems to ensure optimal performance and reliability.
- Develop and maintain documentation for data engineering processes and systems.
- Implement data governance best practices for data quality lineage and metadata management.
- Assist in the evaluation and selection of appropriate GCP services for specific data processing needs.
- Stay updated with GCP developments and recommend innovative solutions to improve data engineering processes.
Required Qualifications:
- Bachelors or masters degree in Computer Science Data Engineering or related field.
- Proven experience in designing and implementing data processing systems on Google Cloud Platform.
- Proficiency in programming languages such as Python for data processing and ETL (Extract Transform Load) tasks.
- Strong understanding of big data technologies and frameworks.
- Experience with GCP services such as BigQuery Dataflow Pub/Sub and Dataproc.
- Expertise in SQL and database technologies for data manipulation and querying.
- Ability to troubleshoot and optimize data processing workflows for performance and costefficiency.
- Excellent communication skills and the ability to collaborate in crossfunctional teams.
- Understanding of data governance principles and best practices.
- Familiarity with machine learning pipelines and model serving on GCP is a plus.
- Certifications in GCP data engineering or related areas are preferred.