Senior Data Engineer

Senior Data Engineer

Responsibilities:

DATA ENGINEERING

Use Airflow for workflow orchestration to create ETL pipelines that ingest application loan fraud payment model data from disparate data sources and validate end product
Use data modeling techniques to store above data in an AWS S3 Data Lake
Collaborate closely with data science to design new models aggregating and validating different features and targets
Leverage Trino and Pandas for comprehensive analysis of application loan fraud payment and model datasets delivering critical insights
Work with data vendors to provide additional info for segmenting/labeling
customers/apps
Pull customer/application data for data retros or debt sales

SOFTWARE ENGINEERING

Implement XGBoost models in Python
Design and implement new underwriting policies in Python
Design and implement backend features in Django REST framework to support customer segmentation A/B testing and selection of underwriting strategies
Design and implement decision models using Python in a Django framework
Integrate new data vendors into the Django web server designing and constructing robust data models
Adapt API request and response structures to accommodate evolving requirements
Collaborate with external partners to facilitate the implementation of models and ensure their accuracy through comprehensive validation
Unit/end to end testing of models and APIs

DEVOPS/SITE RELIABILITY

Use Opensearch Slack alerts AWS SES (email) and AWS Cloudwatch alerts to monitor the following systems: DecisioningEngine Modelhoster Airflow and UOwnCoreLambda
Design and implement API endpoints utilizing AWS CDK and AWS Lambda
Build and maintain CDK AWS CodePipelines for CI/CD
Maintain AWS Opensearch service. Write queries for aggregating underwriting data used for debugging and alerting purposes
Deploy and maintain following docker swarm clusters: DecisioningEngine and
Modelhoster
Maintain/deploy MySQL DB servers: DecisioningEngine and Modelhoster
Maintain/deploy PostgreSQL DB servers: Zoca
Maintain/deploy various EC2 containers
Debug/monitor website/backend logs using CloudWatch
Design and build docker containers for various services


MISCELLANEOUS

Maintain/extend single page application built with Node.js React MobX and SCSS code
Maintain/extend landing page built with Python and Django framework
Maintain/extend web server built with Python Sanic framework
Answer ad hoc requests for audits by examining various data stores (Opensearch Postgresql MySql Cloudwatch logs etc) sometimes writing Python scripts to pull bulkdata
Write python scripts to interface with LMS via SOAP based APIs or chromium scripts
Automate underwriting tasks using Docker Bash and Python such as settling loans or pulling delinquent loans within certain time windows etc
Reach out to customers via CampaignMonitor and Twilio for marketing or loan settlement purposes
Work with data vendors to identify new rules/triggers to stop frau

Remote Work :

No