A lot of people have hit on reasons that I think are valid - it wasn't just one thing. But I think you also have to consider that every trijet was its own unique animal. They weren't all retired en masse, they were retired one by one, for their own reasons, but usually because a more efficient replacement came along. And in the case of existing trijets, those replacements came in the form of twinjets, which are pretty much always going to be more efficient.
That recalibrates the question a little bit to "why didn't new trijets replace old quads?" And the answer there is still mainly "because those quads were replaced with twins", although in some cases, they were replaced with newer quads either because four engines were needed for performance reasons (the A380) or because it was cheaper to adapt an existing design (the later 747 variants) than to create a new twin of the same size.
That just leaves the loose end of why those individual trijet designs were each dead ends while we're still getting updated 737's and 747's, and that question is covered by some of the earlier answers regarding maintenance and other things. One thing I've not yet seen mentioned is that the larger trijets often had/have weight restrictions on various runways at various airports because of that rear center-mounted engine, which concentrates the weight in the center of the airplane. I'm not sure if this could have been overcome in time with different gear designs, but I know that at least at some airports, a much heavier 747 could land with no restrictions whereas an L-1011 could only land at something like 511,000 lbs. or less. And this is also why the later DC-10 variants had that center gear, to spread and carry the weight a bit more evenly. So this is at least a concern with trijets that doesn't exist with quads or twins.
That makes no sense, the entire
weight of an aircraft is transmitted
to the ground through its landing gear
No matter where that weight is situated.
Now certain airports have pavement
restrictions that limit how much weight
can be imposed by each wheel or axle
That would explain lower limits for some
L1011 models with its twin bogie gear
configuration at a few airports
(LGA was one )
The 747 with its quad bogies and weight
spread over sixteen main wheels and two
nose wheels has a big advantage here
The heaviest DC10’s were quite a bit heavier than even the -500 version of
the Tristar and needed that center
gear to spread the load
It had nothing to do with engine
The best contribution to safety is a competent Pilot.