Senior Data Engineer Only W2
Senior Data Engineer Only W2
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Role : Senior Data Engineer
Location : Quincy MA
If interested in below JD Please share your resumes to
As a Senior Data Engineer CASM Platform you will:
- Data Integration API Development: Integrate diverse cybersecurity data sources using varietyof API mechanisms and to standardize and streamline data across the data and user planes.
- Build and maintain data APIs for seamless access to data pipelines enabling realtime insights for applications machine learning models and analytical layers.
- Data Pipeline Engineering Optimization: Design develop and optimize largescale ETL/ELTpipelines on Databricks to efficiently process and transform cybersecurity data. Utilize PythonPySpark and Databricks to automate and standardize data workflows across stages (rawcleaned curated) ensuring scalability and high performance.
- Data Quality & Governance: Implement automated data quality checks leveraging DatabricksDQM tools and CI/CD pipelines to uphold data integrity and governance standards. Ensure datlineage metadata management and compliance with cybersecurity and privacy regulationsapplying rigorous quality standards across data ingestion and processing workflows.
- Data Analytics & Visualization: Design centralized data models and perform indepth data analysis to support cybersecurity and risk management objectives. Develop visualizations anddashboards using tools like Databricks encapsulate data to spin up to React.js application layerto provide stakeholders with actionable insights into threat landscapes vulnerability trends andperformance metrics across the platform.
- Scalable & Secure Data Architecture: Architect and manage secure highperformance dataenvironments on Databricks utilizing AWS services such as S3 ELB and Lambda. Ensure dataavailability consistency and security aligning with AWS best practices and data encryptionstandards to safeguard sensitive cybersecurity data.
- Agile Product; Engineering Continuous Delivery: Collaborate with advanced Agile Product; Engineering crossfunctional teams to deliver datadriven insights through analytics tools andcustom visualizations that inform strategy and decisionmaking. Empower stakeholders withtimely actionable intelligence from complex data analyses enhancing their ability to respond toevolving cybersecurity risks.
- Data Science & ML Integration: deploy machine learning models including predictive analyticsanomaly detection and risk scoring algorithms into the CASM platform. Leverage Python andPySpark to enable realtime and batch processing of model outputs enhancing CASM Platforms proactive threat detection and response capabilities.
- Mentorship; Best Practices Promotion: Mentor junior engineers establishing best practices in data engineering DevOps data science and analytics. Encourage high standards in modeldeployment data security performance optimization and visualization practices fostering aculture of innovation and excellence.
Education Qualifications:
Minimum Qualifications
Education: B.S. M.S. or Ph.D. in Computer Science Data Science Information Systems or a
related field or equivalent professional experience.
Technical Expertise: 8 years in data engineering with strong skills in Python PySpark SQL
and extensive handson experience with Databricks and big data frameworks. Expertise in
integrating data science workflows and deploying ML models for realtime and batch processing
within a cybersecurity context.
Cloud Proficiency: Advanced proficiency in AWS including EC2 S3 Lambda ELB and
container orchestration (Docker Kubernetes). Experience in managing largescale data
environments on AWS optimizing for performance security and compliance.
Security Integration: Proven experience implementing SCAS SAST DAST/WAS and secure
DevOps practices within an SDLC framework to ensure data security and compliance in a high
stakes cybersecurity environment.
Data Architecture: Demonstrated ability to design and implement complex data architectures
including data lakes data warehouses and lake house solutions. Emphasis on secure scalable
and highly available data structures that support MLdriven insights and realtime analytics.
Data Quality ; Governance: Handson experience with automated data quality checks data
lineage and governance standards. Proficiency in Databricks DQM or similar tools to enforce
data integrity and compliance across pipelines.
Data Analytics; Visualization: Proficiency with analytics and visualization tools such as
Databricks Power BI and Tableau to generate actionable insights for cybersecurity risks threat
patterns and vulnerability trends. Skilled in translating complex data into accessible visuals and
reports for crossfunctional teams.
CI/CD and Automation: Experience building CI/CD pipelines that automate testing security
scans and deployment processes. Proficiency in deploying ML models and data processing
workflows using CI/CD ensuring consistent quality and streamlined delivery.
Agile Experience: Deep experience in Agile/Scrum environments with a thorough
understanding of Agile core values and principles effectively delivering complex projects with
agility and crossfunctional collaboration.
Employment Type
Remote
Company Industry
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
