From Ayer's Cliff you are near the border of Vermont! If you are into skiing Jay's Peak in Vermont is a very short drive and has great slopes. There is also Mont Orford in Quebec and several other ski hills all within an hour's drive. I would also check out North Hatley.
For shopping Sherbrooke and Magog are short drives away and both have a lot to offer!.