BigData Integration Consultant

Key Responsibilities:

1. Data Ingestion and Integration:
   • Develop and maintain data ingestion processes to collect data from various sources.
     
   • Integrate data from different platforms and databases into a unified data lake.
     
2. Data Processing:
   • Create data processing jobs using Hive and PySpark for largescale data transformation.
     
   • Optimize data processing workflows to ensure efficiency and performance.
     
3. Data Pipeline Development:
   • Design and implement ETL pipelines to move data from raw to processed formats.
     
   • Monitor and troubleshoot data pipelines ensuring data quality and reliability.
     
4. Data Modeling and Optimization:
   • Develop data models for efficient querying and reporting using Hive.
     
   • Implement performance tuning and optimization strategies for Hadoop and Spark.
     
5. Data Governance:
   • Implement data security and access controls to protect sensitive information.
     
   • Ensure compliance with data governance policies and best practices.
     
6. Collaboration:
   • Collaborate with data scientists analysts and other stakeholders to understand data requirements and provide data support.
     

Qualifications:

• Bachelors degree in Computer Science Information Technology or a related field.
  
• 3 years of experience in big data engineering and data processing.
  
• Proficiency in Hive Hadoop and PySpark.
  
• Strong SQL and NoSQL database experience.
  
• Experience with data warehousing and data modeling.
  
• Knowledge of data integration ETL processes and data quality.
  
• Strong problemsolving and troubleshooting skills.
  
• Excellent communication and teamwork skills.
  

Preferred Qualifications:

• Experience with cloudbased big data technologies (e.g. AWS EMR Azure HDInsight Google Dataprep).
  
• Certification in Hadoop Hive or PySpark.
  
• Familiarity with containerization and orchestration tools (e.g. Docker Kubernetes).
  
• Knowledge of data visualization tools (e.g. Tableau Power BI).