SW Harbor IS nicely off the beaten path
We were there just after Memorial Day this year. All of Mt. Desert (pronounced De-ZERT) Island is less busy and crowded that early in the season, but it was still more peaceful than Bar Harbor, which was pretty active even then. The locals say it never gets too crowded or busy there, but there were enough people around that it did not feel deserted (excuse the pun). It is centrally located to island sites and activities and, from late June to early October, the Island Explorer free shuttle buses come through, so you do not even have to drive. There are quite a few nice-looking B&Bs and rental properties there and everyone was very nice.