According to a USAF study, based on an analysis of a Titan IV* explosion shortly after liftoff in 1998, the resultant extremely large fireball from the solids, punctuated by chunks of burning solid rocket fuel, would make the Orion escape system fail if used during the first minute of flight. As the Orion would still be inside the fireball when it deployed its parachutes (and would remain so until it impacted the ocean if it delayed deploying it chutes), the parachutes would be fatally damaged.
A significant part of the problem is the very high initial acceleration of Ares I, which causes very high dynamic loads early in the flight, which significantly reduces the escape system's rockets ability to pull (or push) the Orion away from the booster. The other significant part is that the SRB, being solid, cannot practically be shut down before burnout. A significantly more powerful escape rocket would solve the problem, but at the obvious costs in weight for the rocket and added structure. It's also unclear that the escape system's acceleration could be increased significantly, due to limits of human G tolerance, which makes the size increase worse.
Ares I / Orion is already having severe weight problems.
Some are calling this issue fatal for Ares I (obviously premature, but anything that adds more weight growth to the stack at this point is very troubling), and NASA states that its analysis is that this is not an issue, but they will be looking at the USAF's study.
*The Titan IV has a pair of large SRBs – each roughly half the size of the Shuttle/Ares SRBs