No the biggest problem with durians is not the odor, although that seems to be the only problem ever discussed. The smell discussion has become so much of an obsession that it seems to me the real bigger problems are never even brought up.
For example,this is probably the least user friendly fruit imaginable. The whole outer rind is covered in needle sharp spines. Carrying one of these fruits is hard enough. Opening them up can be right down dangerous. Some cultivars are easier to open than other. Some will pop open at the perfect ripe stage. The 2 Thai varieties i'm dealing with now are over ripe once the segments pop open. So prying them open is quite a strong laborious effort and takes a lot of skill to open one without mauling your hand.
Then there is the problem of lots and lots of sharp rind waste after eating to dispose of. One past calculation i did on a Chanee showed that 72% of the fruit is waste and only 28% is pulp. http://tropicalfruitforum.com/index.php?topic=13525.msg171416#msg171416
If you are buying the fruits for $5 a pound then you are in fact paying around $17.85 for the edible portion. So it's a very expensive fruit!
From a cultivation point of view the plant material is hard to obtain, hard to transport, scions stay viable a very short time and are hard to graft. The trees are slow to bare, often taking as long or longer than the notoriously slow mangosteen. The trees get quite large and the wood is brittle and breaks easily in high winds. The trees easily succcumb to phytophthora (fungal root rot).
So why do people bother? First taste of an exquisite durian and all these problems don't seem very problematic any longer. As a fruit durian is really in a class all its own.