In VT you could hike the Long Trail in the region of the Big Branch Wilderness Area. Plenty of brooks, the Big Branch itself, and some ponds along the way like
Little Rock Pond which is stocked with brook trout.
In NH/ME many people do fishing trips along the Androscoggin River; however, these tend to be on canoe or kayak, not on foot. In the Whites your best bet would be the Pemi Wilderness area, though be aware of the wilderness regs before going in there. I know Lincoln Brook has trout (a friend has pics of them jumping) and Franconia Brook likely does as well, but no promises.
The Wild River, up in the new Wild River Wilderness Area, is another option. Neither of these will really give you 4 days, however. 2 or 3 at best, I would think.
Baxter has lakes and ponds all over the place. An excellent suggestion!