What is important to Boeing's decision is not the number of A380 orders but how it does in airline service. Boeing looks not at current orders, but what future orders are likely - considering passenger demand, oil prices, etc. Some half-educated guesses.....
.........If the plane is pretty good but airlines have trouble filling the seats profitably, Boeing will not build a competitor. This was the case for the 747 for a very long time, and that is one reason it has not had a true competitor until recently.
.........If the plane is sort of a dog, like the MD
-11, but airlines have no trouble filling the seats profitably, a competior will be built in short order.
.........If the plane is good and ailines have no trouble filling the seats, a competitor may eventually be built but other priorities (such as a 737 replacement) will come first.
........If the plane is a dog and the airlines can't fill the seats, Boeing will probably do no more than a conservative update of the 747 or 777, in the hopes that their plane will be small enough and good enough to be a better competitor. But it is harder to guess what they will do in this situation than in the others. They could just get complacent and sit pretty. That is what they have done in the past when they were ahead. The modern Boeing company does not like to innovate unless absolutely forced to. It doesn't the next quarterly earnings report. Don't worry, Boeing fans. Airbus is probably going to also be more like this in the future now that its product line is complete.
Boeing argues that until recently, MOST(highlighted because some in this forum don't like to hear this word. It prevents them from ruining discussions.) passenger 747's were bought more for range than capacity. Once planes appeared with the same or better range and a little less capacity, such as the newer -B and -C market 777's and A340's, the 747 was largely abandoned as a new build passenger airplane. Airbus argues that this was due to newer technology on the newer designs. Probably both are right to some degree.
As for the DC-10/L1011 comparison - the DC-10 succeeded technically. It also succeeded in sales. It did not succeed in profit. That is what matters in the commercial world. Products with DIRECT competitors are far less likely to be profitable than products without such competitors. Airbus and Boeing, as well as the RJ
makers, avoid direct competition in their development programs when they can.