Don't remember the exact reason for the cancellation of the 764ERX, but would assume it was due to the overall poor sales performance of the 764, and the cost of developing the ERX for a relatively small market.
The relatively poor takeoff performance of the 764 is probably due to the fact that to save costs the same engines as the 763 are used, despite the 764's higher weights (The engine is the: GE CF6-80C2B7F)
The 764ERX addressed that issue by offering the much more powerful RR Trent or PWGE Alliance engine to cope with the increased weights of the longer-range model.
Unfortunately, I think the cancellation of the 764ERX sealed the success of the A330-200 and the fate of the B767-400, and handed this market segment to Airbus. Since it's launch the 767-400 has had no customers, and the likes ILFC and Kenya Airways have cancelled. The A330-200 has enjoyed continued success, no doubt aided by the lack of direct competition from Boeing.
I think Boeing may have fallen into the same trap British aircraft makers in the late 1950s/early 1960s, when the VC-10 was designed to meet BOAC requirements and the Trident was designed for BEA. Because these aircraft were so airline-specific, their sales records to other carriers was abysmal. In the same way the 767-400 was designed to stop Continental/Delta defecting to Airbus and meet the needs of US Domestic carriers to replace their DC-10/L1011 fleets. It meets those requiements well but the cancellation of the 764ERX, which would have made the 764 very attractive to potential customers, means the 767-400 is never going to have much appeal outside the US - most potential customers choosing the A330/B777 or even sticking to the B767-300 with it's superior range. Range usually seems to be the reason protential 764 customers look elswhere.
Shame really, I flew on a Delta B767-400 last year, and it was a very nice aircraft, certainly from a passenger perspective the new cabin is a considerable improvement over the old B767-2/300.
View Large View Medium
Photo © PeachAir