There is a simple, but very efficient application available called “Xenu's Link Sleuth”. It can crawl site fast, but may not able to simulate different clients though.

http://home.snafu.de/tilman/xenulink.html