Airlines have tried it before on a small scale. From the airline's point-of-view you would want an auction where people bid a maximum price and when the plane fills up people start to get bumped off based on their maximum bid, but I think the concept is not one that the customer would really understand or like because you could not guarantee your seat very easily.
So, it could work like this:
JFK-LAX; 100 coach seats to sell at 0930 on May 16th; Minimum Bid: $25
Customer bid/maximum bid:
Now 99 other people bid. When the 101st person bids the lowest price maximum bidder is bumped. Theoretically this could be workable if it was only part of the plane AND
you stopped selling seats at 21 days prior so people would no know whether they were going or not. The other problem is that if the flight were being sold elsewhere via normal channels it would create a problem as people would stop bidding and go over there if it were cheaper there.