Now here's a BIG reason. In the west, the availability of private jet aviation at reasonable costs and increased ranges have made the super expensive cabins within America and to Europe from America less fillable with full fare customers.
But it's still not practical to send shared private jets between the US/Europe and Asia and the south pacific/australia regions on a regular basis. thus the first class luxury product is still vital. Add to that the longer time you spend on the plane, and you get more of an answer.
Also, businesses are not sending their people first class like they used to. Remember that business class now offers first class or better comfort from 20 years ago. If it was good enough for top employees then, then a fully flat business product should be good enough now, no? So the First cabin in these situations is for the really wealthy, who again, use private jets these days whenever possible.
You might also combine that with the larger percent of US and Europe populations who can afford to fly, and that flying for them isn't a luxury for the wealthy. Since this is the case, as a percentage of the flying public, most western flyers are not accustomed to being pampered and are willing to sit in cattle car class, or at least business class, for a longer amount of time. Just look at inter-Europe flights. No first class, business is just quicker boarding and your own armrest and is flexible based on demand, but very little added luxury, because Europeans don't see the value. In America, the first class domestic cabins are smaller than they were for the same reason, and are more perks for frequent flyers as much as anything.
I think if you add these three things up with other factors, you see what is likely the cause.
Of all the things to worry about... the Wookie has no pants.