SDE

Key Responsibilities:

• Develop maintain and optimize web scraping tools to extract large volumes of data from multiple websites and APIs.
• Design and implement scalable data pipelines for efficient data collection storage and processing.
• Write clean efficient and maintainable code for data scraping and parsing.
• Implement solutions for dynamic website scraping handling AJAX requests Captchas and other challenges.
• Ensure data quality by validating cleaning and formatting scraped data.
• Collaborate with crossfunctional teams to understand data requirements and integrate data into the companys data infrastructure.
• Monitor and troubleshoot scraping jobs ensuring timely and accurate data collection.
• Stay uptodate with the latest trends and technologies in web scraping automation and data extraction.

Required Skills and Qualifications:

• Bachelor s degree in Computer Science Engineering or a related field.
• 2 years of experience in data scraping web crawling or data extraction.
• Proficiency in programming languages like Python Java or Node.js.
• Strong knowledge of libraries and frameworks such as BeautifulSoup Scrapy Selenium Puppeteer or Playwright.
• Experience working with APIs (RESTful GraphQL) and understanding of HTTP/HTTPS protocols.
• Handson experience with database systems (SQL NoSQL) for storing and processing large data sets.
• Familiarity with cloud services like AWS GCP or Azure for scaling data scraping solutions.
• Experience handling dynamic web pages Captchas and ratelimiting challenges.
• Solid understanding of data structures algorithms and software design principles.
• Strong debugging problemsolving and optimization skills.

Preferred Qualifications:

• Experience in building scalable distributed data scraping systems.
• Familiarity with DevOps practices CI/CD pipelines and version control (Git).
• Exposure to big data tools such as Hadoop Spark or Kafka.
• Knowledge of web automation and bots development.

programming languages,devops practices,web automation,data,web scraping,data extraction,data scraping,apis,cloud services,big data tools,database systems,libraries and frameworks