PDA

View Full Version : SOAK: - Web-Crawling/Monitoring software or service


Makita
10-01-2008, 12:20 PM
Anyone know of a software or service that one can use for crawling websites?

@ work we need to monitor our merchants with web-sites (whether they sell items there or not) to look for changes, updates, etc. Taking screenshots, or prompting a human to review manually, as it went.

We have a company available, but even with discounted pricing, it is $10K per year, and we don't really have enough internet-based merchants to warrant this cost.

Otto
10-01-2008, 02:27 PM
How you do it depends on how the websites work. I mean, many sites change every time you look at them, dynamic content is commonplace.

Why not subscribe to their feeds, if they have them? Or run wget on a cron job and then diff against the old site. Lots of ways to do it, but they'll all be fairly custom.

Makita
10-01-2008, 06:13 PM
The more we think about it, we need a service, doubt there is software that can do this easily and there only seems to be 1 company offering it. I can see M/C's logic here, but really not sure it is feasable in the long-term. (we basically need to touch every link on the site and document where it goes AND that is isn't selling something it shouldn't).

dansee
10-01-2008, 08:32 PM
In fairness, for what you're asking, the price doesn't seem too steep.

JYoung
10-01-2008, 11:59 PM
http://members.dslextreme.com/users/jonathany/images/icons/sign_google.gif





;)