Why Vermont over other parts of New England? What does it offer that I can't get elsewhere?
If you're after hiking, rock climbing, kayaking, rafting, etc there are much better options in New Hampshire and Maine.
Yes, it is not as crowded. That is one nice feature, but I think that is because it has less to offer. The most visited tourist attraction in the state is the Ben and Jerry's factory. That speaks volumes.