Overview
The Big Data Engineer plays a crucial role in the organization by designing developing and maintaining scalable data processing systems. As companies increasingly rely on datadriven insights to fuel their operations and strategic decisions the importance of robust data engineering cannot be overstated. This position is essential for transforming raw data into meaningful information that can drive business growth and innovation. The Big Data Engineer works closely with data scientists analysts and stakeholders to ensure that the architecture meets business needs and optimizes performance. They are responsible for implementing the latest tools and technologies in big data helping organizations manage analyze and derive insights from large volumes of data across various sources.
Key Responsibilities
- Design and implement scalable data processing systems.
- Develop ETL (Extract Transform Load) pipelines for data ingestion.
- Collaborate with data scientists to optimize data workflows.
- Maintain and enhance big data architectures.
- Ensure data quality and integrity through robust validation processes.
- Monitor data systems for performance issues and optimize appropriately.
- Integrate new data sources into existing systems.
- Utilize Hadoop Spark and other big data frameworks.
- Develop and maintain data modeling solutions.
- Create documentation for data architecture and processes.
- Implement data governance and compliance standards.
- Participate in code reviews and best practices discussions.
- Collaborate with crossfunctional teams to align data initiatives with business goals.
- Perform troubleshooting and debugging for data solutions.
- Stay updated with emerging technologies in big data.
Required Qualifications
- Bachelors degree in Computer Science Engineering or related field.
- 3 years of experience in data engineering or related roles.
- Proficiency in programming languages such as Java Python or Scala.
- Extensive experience with big data technologies especially Hadoop and Spark.
- Familiarity with data warehousing solutions like Redshift or Snowflake.
- Experience with ETL tools such as Apache NiFi Talend or similar.
- Strong understanding of SQL and NoSQL databases.
- Knowledge of cloud platforms such as AWS Azure or Google Cloud.
- Experience integrating data from a variety of sources.
- Understanding of data modeling concepts and techniques.
- Ability to work with large complex datasets and derive meaningful insights.
- Knowledge of data governance frameworks and compliance regulations.
- Strong analytical and problemsolving skills.
- Excellent communication skills both written and verbal.
- Adept at working in a fastpaced and dynamic environment.
java,nosql,redshift,talend,hadoop ecosystem,python,sql,data governance,sql proficiency,apache nifi,scala,hadoop,communication,snowflake,performance tuning,aws,etl,problem-solving,data modeling,big data,apache kafka,azure,spark,data warehousing,google cloud