SOAK: - Web-Crawling/Monitoring software or service

[Makita]

Anyone know of a software or service that one can use for crawling websites?

@ work we need to monitor our merchants with web-sites (whether they sell items there or not) to look for changes, updates, etc. Taking screenshots, or prompting a human to review manually, as it went.

We have a company available, but even with discounted pricing, it is $10K per year, and we don't really have enough internet-based merchants to warrant this cost.

[Otto]

How you do it depends on how the websites work. I mean, many sites change every time you look at them, dynamic content is commonplace.

Why not subscribe to their feeds, if they have them? Or run wget on a cron job and then diff against the old site. Lots of ways to do it, but they'll all be fairly custom.

[Makita]

The more we think about it, we need a service, doubt there is software that can do this easily and there only seems to be 1 company offering it. I can see M/C's logic here, but really not sure it is feasable in the long-term. (we basically need to touch every link on the site and document where it goes AND that is isn't selling something it shouldn't).

[dansee]

In fairness, for what you're asking, the price doesn't seem too steep.

[JYoung]

http://members.dslextreme.com/users/jonathany/images/icons/sign_google.gif





;)