Leaving Home - I like to leave early in the morning. I don't like to sit around the house and wait - I like to feel that I didn't waste my first day of the trip at home. Plus, I like to get to my destination when there is still a reasonable amount of daylight, (especially if it's a place where I've never been before) because it's easier to find your way around if you're not farmilliar with the area.
Returning Home - I like my vacation to last as long as possible, so I prefer to take the last flights of the day. (I like to be one of the first to give up my seat if the flight is overbooked.) I like to arrive around 9, 10, or 11 at night. Those are usually the times I go to bed, so I can just go home, jump in the bed, and worry about unpacking the next day.