There are alot of things i havent tried growing, but from the ones i did the most problematic were :
Lychee- i would have to outlive the planet if i wanted to get to the point of it fruiting, such a slow grower here, from seed.
Papaya- very fussy about soil moisture in my climate.
Cacao- ive tried several times to germinate fresh seeds, right from a cacao pod. The problem is its awlays available in the depth of winter, and such tropical plant seeds are hard as a rock to germinate with my inconsistent temperatures in home.
Longan- same as lychee
I would also have to add citrus to the most difficult for me, i just cant make them happy , awlays some kind of defficiency, too much moisture or whatever , very hard for me to grow. On the other hand i have a friend which growns several citrus plants in pots, she is doing an amazing gob with them, compared to me haha.
I also kinda strugle with plants that dont want alot of moisture, cacti and others. Whille i do have nice dragon fruits growing in my gh ,i awlays wonder if i'm waterring too much or too little, regarding all the gh plants.