For overall energy saving in a home, the highest priorities are air-tightness of any duct systems and air-tightness of the thermal boundary surface of the home.
Using the Blower Door and Duct blaster we depressurize the home or duct system to measure the rate of air leakage and identify sources of leakage. This works great in conjunction with IR imaging to see cold air infiltration through walls and ceilings or conditioned air escaping from ducts. You can also easily feel air leakage with your hands when you know where to look. A particular area of interest is outward leakage from the upper floors of a house. Due to hot air rising, normally air leaks outwards from the top parts of a home. This means occupants don't feel the leaks because cold air doesn't normally come in here. But cold air to replace this lost warm air is being pulled in through the bottom of the home. [In an multi-story apartment or condo building, in mid-winter, upper level residents can leave their window open and not feel much effect - indeed they might not need to heat the air in their apartment at all. However, this increases drafts into the units at the bottom of the building. The residents at the bottom of the building have to heat their units more and that warm air quickly escapes to the lucky occupants of the units at the top of the building! Air sealing between the units of a multi-family can help stop this effect.]
Worried about your home being too tight? Here's a very thorough explanation of this concern.
Does your house have to breathe? No, but people do, and anything that gets wet needs to be able to dry out. Homes produce moisture from many sources: showers, cooking, cleaning, etc. If the air is not well ventilated, and poorly insulated exterior surfaces are cold, condensation will occur. If, however, exterior walls are insulated correctly and the air barrier (typically the drywall) is in complete contact with the insulation, and there is no vapor barrier trapping warm moist air on a cold surface, then the wall insulation will work well and there will not be moisture problems from condensation. For Energy Star Homes we cover all of this and more with the thermal bypass checklist.