author thumb

Hello, my name is Zhewei Hu
Incoming SRE@Pinterest |
Ph.D. student@NCSU |
SRE Intern’18@Procore |
SDE Intern’17@IBM |
SDE Intern’16@Offers.com

Web Crawler Application – Scraping

Description

Based on a customer’s need - car model, year of make, mileage, we find the posts on craiglist, which agree with the customer’s need and look up the price for the car on sale on the post from Kbb. Then our website will show the customer the posts (link to the original posts on craiglist) with cars he is interested in on sale and the price from Kbb. Right now, people have to go to craiglist to find the car they are interested in with a seller’s price, and then go to kbb, checking out the review price. Our website will do the two jobs for the customer automatically, and people can get updated information from our website’s subscription service.

Workflow

ProjectScraping--design

Conclusion

  1. Web crawler is a powerful tool. It will save a lot of time when searching information from complicated website.
  2. Some websites will block commercial servers from crawling their websites.
  3. The legality of scraping, especially for commercial use is not clear.

/end