Let me list several reasons:
Fears of terrorism, especially as Greece is relatively close to the middle-east and the war in Iraq.
High costs of tickets in a place with a modest incomes of it's own people.
Small national population.
Isolation by land to most of Europe, thus discouraging travel by train or car from adjacent countries.
Held in August, so vacations in most places in the world, including the host country
Extemely hot weather (often in high 30's C/high 90's F during the day).
Fears of venues not being ready.
Fears by Americans of being subject to terror attacks, or fears of having to open their ears and listen to some critizism of USA policy.
Improvements in TV
coverage all over the world, with each countries' coverage tailored to the sports most linked to their country or region of the world or possible medal winning athletes. Why go to the games if you can see them on TV
High costs of airfares, accomidations, as too high of expections of demand and as peak summer travel months.
Other local/regional/sports distractions, like NASCAR racing in the USA.
People don't have the time to travel then.
Overall less interest in the Summer Olympics as is summer and vacation time in the Northern Hemisphere, so more interested in being outside, rather than watching TV
indoors. Also, many of the events are not something people are particulary interested in.