Senior Data Engineer/Senior data engineer (Data mining)

Job Overview: Sproutsai is seeking a highly skilled and experienced Senior Data Engineer/Scraping System Developer with a minimum of 7 years of hands-on experience in designing, developing, and maintaining production-grade scraping systems. The ideal candidate will be responsible for leading and implementing robust scraping solutions that align with our business objectives.

Responsibilities: Lead the design and development of scalable, reliable, and efficient scraping systems to gather and process data from various online sources. Collaborate with cross-functional teams to understand data requirements and implement scraping solutions that meet business needs. Evaluate and select appropriate scraping tools and technologies, considering factors such as performance, scalability, and maintainability. Implement data quality and validation processes to ensure the accuracy and integrity of scraped data. Monitor and optimize scraping systems for performance, reliability, and resource utilization. Stay current with industry trends, emerging technologies, and best practices in web scraping and data engineering.

Qualifications: Bachelor's or Master's degree in Computer Science or a related field. Minimum of 7 years of professional experience in designing and implementing production-grade scraping systems. Expertise in web scraping frameworks, tools, and techniques. Strong programming skills in Python or Java. Experience with distributed computing and parallel processing for large-scale data scraping. Knowledge of data storage and database technologies (e.g., SQL, NoSQL). Familiarity with API integration and data extraction from various sources. Proven track record of delivering high-quality solutions on time and within budget.

Additional Skills (Preferred): Familiarity with ethical scraping practices and compliance with legal requirements. Experience with machine learning and natural language processing for data extraction. Knowledge of data security and privacy considerations in scraping systems.