Senior Big Data Engineer

We are seeking a talented Big Data Engineer with 2 to 5 years of experience to join our data engineering team. The successful candidate will play a crucial role in designing building and maintaining our big data infrastructure and pipelines. You will work with large datasets ensuring data accessibility reliability and efficiency to support datadriven decisionmaking processes across the organization.

Key Responsibilities:

• Data Pipeline Development:

  • Design develop and maintain scalable data pipelines to process large volumes of data efficiently.
  • Implement robust ETL (Extract Transform Load) processes to ensure data integrity and quality.

• Data Storage and Management:

  • Manage and optimize big data storage solutions (e.g. HDFS NoSQL databases) for performance and scalability.
  • Develop and maintain data warehouses and data lakes to support analytical and reporting needs.

• Data Processing:

  • Utilize big data processing frameworks such as Apache Spark Hadoop and Kafka to perform complex data transformations and analytics.
  • Develop both realtime and batch data processing solutions to meet business requirements.

• Data Integration:

  • Integrate data from various sources including structured and unstructured data to create unified datasets.
  • Collaborate with data scientists and analysts to ensure seamless data integration for analytical purposes.

• Performance Optimization:

  • Optimize data processing performance and efficiency through performance tuning and optimization techniques.
  • Implement best practices for data partitioning indexing and compression to enhance query performance.

• Data Quality and Governance:

  • Establish and enforce data quality standards and data governance policies to ensure data accuracy and compliance.
  • Implement data validation and monitoring processes to identify and address data quality issues proactively.

• Collaboration and Leadership:

  • Work closely with crossfunctional teams including data scientists analysts and software engineers to understand data requirements and deliver effective data solutions.
  • Provide technical leadership and mentorship to junior team members fostering their professional growth and development.

• Continuous Improvement:

  • Stay abreast of emerging technologies and industry trends in big data and data engineering.
  • Drive continuous improvement initiatives to enhance the efficiency scalability and reliability of our big data infrastructure and processes.

Required Qualifications:

• Education:

  • Bachelor’s degree in Computer Science Information Technology or a related field.

• Technical Skills:

  • Proficiency in big data technologies such as Hadoop Spark Kafka and related ecosystems.
  • Strong programming skills in languages like Java Scala or Python for data processing and scripting.
  • Experience with ETL tools and processes for data integration and transformation.
  • Solid understanding of SQL and NoSQL databases for data storage and retrieval.
  • Familiarity with cloud platforms (e.g. AWS Azure Google Cloud) and their big data services.

• Experience:

  • 2 to 5 years of handson experience in big data engineering roles with a proven track record of delivering complex data solutions.

Soft Skills:

• Strong problemsolving and analytical abilities.
• Excellent communication and collaboration skills.
• Ability to work effectively in a fastpaced and dynamic environment.
• Attention to detail and a commitment to delivering highquality solutions.