Data Engineer
Data Engineer
Posted on :
16-10-2024
Employer Active
1 Vacancy
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Share
Send me jobs like this
Job Alert
You will be updated with latest job alerts via emailValid email field required
Send jobs
Job Description
Responsibilities:
- Implementing and designing scalable optimized data pipelines for (pre) processing ETL for machine learning models.
- Develop and maintain conceptual and logical data models using data modeling guidelines from the clients;
- Document and maintain business glossary in the enterprise data catalog solution;
- Evaluate business data models and physical data models for variances and discrepancies;
- Support project team in adopting business data models;
- Guide project team to map physical data models to business glossary.
Knowledge/Experience:
For senior experience:
- Handson technologies and frameworks used in ML like sklearn MLFlow TensorFlow;
- Building complex data pipelines e.g. ETL;
- Experience working in cloud environment data cloud platforms (e.g. GCP);
- Understanding of code management repositories like GIT/SVN;
- Familiar with software engineering practices like versioning testing documentation code review;
- Experience with Apache Airflow;
- Experience in setting up both SQL as well as noSQL databases;
- Experience with monitoring and observability (ELK stack);
- Deployment and provisioning with automation tools e.g. Docker Kubernetes Openshift CI/CD;
- Knowledge of MLOps architecture and practices;
- Relevant work experience in ML projects;
- Knowledge of data manipulation and transformation e.g. SQL;
- Setting up/troubleshoot SQL and NoSQL databases.
For medium experience:
- Design and Develop Data Pipelines: Create efficient and scalable data pipelines using GCP services such as Dataflow (Apache Beam) Dataproc (Apache Spark) and Pub/Sub;
- Data Storage Solutions: Implement and manage data storage solutions using GCP services such as BigQuery Cloud Storage and Cloud SQL;
- Data Analysis and Reporting: Optimize SQL queries for data analysis and reporting in BigQuery.
Employment Type
Full Time
Key Skills
Report This Job
Disclaimer: Drjobpro.com is only a platform that connects job seekers and employers. Applicants are advised to conduct their own independent research into the credentials of the prospective employer.We always make certain that our clients do not endorse any request for money payments, thus we advise against sharing any personal or bank-related information with any third party. If you suspect fraud or malpractice, please contact us via contact us page.
