Spark Engineer

Spark Engineer

Contract

Rayleigh NC

Solid resume / With all details only genuine profile

Initial remote After 1 month onsite

PRINCIPAL SOFTWARE ENGINEER

The Role

UAP would be the next generation platform to manage acquisition as a service through low code and config driven architecture.

UAP will also provide

• Configuration driven feed acquisition
• Configuration driven transformation of vendor data into canonical format
• Capability to compare data across vendors and generate gap reports
• Registry to maintain feed metadata
• Lineage to track usage
• Configuration driven distribution

As part of UAP (Unified Acquisition Platform) team you will be working on enhancing the next generation in-house acquisition platform using cloud technologies. UAP will be built using Java Spark and Python programming language and use AWS EMR as the infrastructure for execution. You would be developing lambda, Spark, Python utilities for managing end to end feed flow.

As part of UAP we will focus on building tools and capabilities to accelerate acquisition, compare data across vendors, externalize rules for data conversion, build inventory/registry of feeds and track lineage.

• Develop new application for managing feed inventory and tracking their usage
• Developing application interface using Java Spark for converting vendor feeds into business-friendly format
• Develop accelerators for new feed acquisition
• Develop accelerators for data comparison reports
• Developing Lambda triggers for event driven and batch driven use case
• Supporting production during warranty period
• Participate in code review and define best practice
• Manage branching and deployments without conflicts
• Provide L3 production support
• Enhancing framework for future needs and make it a more generic product.

The Expertise and Skills You Bring

• Bachelor in information technology
• 10+ years of experience with application development and hands on experience with Spark/Scala
• Expertise in providing leadership support by providing technical solutions
• Expertise in handling huge dataset
• Expertise in code reviews for standards and efficient coding practices
• Expertise in building monitoring and alerting tools to report failure or potential performance degradation
• Experience with TDD (Test driven development) to build unit testing and robust regression test suite