Implementing Web Scraping for Integrating 1.3 Million MRO
Products
Client Background
- A US Based E-Commerce company wanted to Integrate a comprehensive database of MRO products into their ecommerce platform which includes
- The customer wanted to extract a large dataset of 1.3 million products across 35 categories from a target website.
- Each product category had 400 pages of data & The customer wants to cleanse the data by removing more than 30% duplicate products.
- The target website had over 1000 columns, and the customer wanted images to be downloaded and zipped into a single file, with image links included in the output .csv format.
- And also the customer required a cost-effective and optimized solution for data collection and cleansing.
Approach To Solution
- The Outsource Bigdata An AIMLEAP Company team developed a custom-programmed web crawler tailored to the client’s specific requirements.
- Conducted an in-depth analysis of the target website’s structure to identify key data points and relevant pages. Set multiple rules within the crawler to accurately capture product descriptions, specifications, images and all desired data points
- The team automated de-duplication with Python to efficiently remove duplicate products. Developed code to compile and zip images, linking them accurately in the output .csv format.
- The use of custom crawler and automation minimized manual intervention, streamlining data processing for faster, efficient handling and timely delivery of high-quality data
75%
Cost Saving
1.2 million
Records Processed
99%
Above 99% Quality assurance
Quick turnaround
25 business days
