Senior Data Engineer

Required and Preferred Skill Sets:

• 1 -2 years of hands-on experience in AWS - EMR [Hive, Pyspark], S3, Athena or any other equivalent cloud; Ability to solve complex problems
• 1-2 years of hands-on experience Spark Batch Processing and some familiarity with Spark Structured Streaming; Ability to solve complex issues
• 1-2 years' experience working experience with Hadoop stack dealing huge volumes of data in a scalable fashion
• 2-3 years of hands-on experience with SQL, ETL, data transformation and analytics functions; Ability to solve complex problems
• 2-3 years of hands-on Python experience including Batch scripting, data manipulation, distributable packages; Ability to solve complex problems
• 2-3 years' experience working with batch orchestration tools such as Apache Airflow or equivalent, preferable Airflow
• 2-3 years working with code versioning tools such as GitHub or BitBucket; expert level understanding of repo design and best practices
• 2-3 years working with deployment automation tools such as Jenkins and familiarity with containerization concepts such as Docker and Kubernetes
• 2-3 years of hands-on experience designing and building ETL pipelines; expert with data ingest, change data capture, data quality; hand on experience with API development; some exposure to Nifi or Kafka
• 2-3 years designing and developing relational database objects; knowledgeable on logical and physical data modelling concepts; some experience with Snowflake
• Preferred 1+ years of experience supporting Tableau or Cognos use cases
• Familiarity with Agile; working experience preferred