Jasper itself is small and there isn't much to do outside of a brewpub and a couple of other restaurants. As far as a town/city, Banff would by far have more to do.
Spent a weekend in Jasper in April 2013. We were actually able to get 1 day of skiing in at Marmot Basin, then spent the next day driving down to the Columbia Icefield and checking out the scenery at the stops along the way. I'd go back for sure, but it would definitely involve fly fishing.