Big Data Engineer

Job Overview:

We are looking for a Big Data Engineer to create solutions suitable for the job in a real-time, highly data-driven organization and make technical innovations happen. With a range of projects to develop data processing applications, you will have the opportunity to use cutting-edge technology across all stages of the lifecycle.

Responsibilities:

• Collect and process raw data at scale for a variety of projects and initiatives.
• Design and develop data applications using selected tools and frameworks as required and requested for a variety of teams and projects.
• Read, extract, transform, stage and load data to selected tools and frameworks as required and requested.
• Perform tasks such as writing scripts, web sing, calling APIs, write SQL queries, etc.
• Work closely with the engineering team to integrate your work into our production systems.
• Process unstructured data into a form suitable for analysis.Responsible for Hadoop development
• Implementation including loading from disparate data sets, preprocessing using Hive and Pig.
• Scope and deliver various Big Data solutions
• Ability to design solutions independently based on high-level architecture.
• Manage the technical communication between the survey vendor and internal systems
• Maintain the production systems (Kafka, Hadoop, Cassandra, Elasticsearch)
• Collaborate with other development and research teams
• Building a cloud based platform that allows easy development of new applications

Requirements:

• A solid track record of data management showing your flawless execution and attention to detail.
• Strong knowledge of and experience with statistics.
• Programming experience, ideally in Python, Spark, Kafka or Java, and a willingness to learn new programming languages to meet goals and objectives.
• Experience in C, Perl, Javascript or other programming languages is a plus.
• Knowledge of data cleaning, wrangling, visualization and reporting, with an understanding of the best, most efficient use of associated tools and applications to complete these tasks.
• Experience in MapReduce is a plus.
• Deep knowledge of data mining, machine learning, natural language processing, or information retrieval.
• Experience processing large amounts of structured and unstructured data, including integrating data from multiple sources.
• Experience with machine learning toolkits including, H2O, SparkML or Mahout
• A willingness to explore new alternatives or options to solve data mining issues, and utilize a combination of industry best practices, data innovations and your experience to get the job done.
• Experience in production support and troubleshooting.
• You find satisfaction in a job well done and thrive on solving head-scratching problems.

.NET