Finally, after a month of hard work (with the occasional “oops it’s 5AM already?!”), I’m happy to announce that Crawler has reached v1.0. It has not been put in production use yet though, so please help test and use it, and report issues and feedback. :)

Check it out here: https://github.com/fredwu/crawler