I feel your pain, Ryan!
Everyone laughed when I got hooked on tropical fruit trees. Boise Idaho is zone 6, so a greenhouse was the only way to go. I struggle getting things to fruit, and cant put anything in the ground due to cold clay soils and huge tree roots under my greenhouse.
I like to think my trees have it pretty good compared to Florida or the tropics - I can give them the weather they need, and avoid the hurricanes, etc. And greenhouses, as a hobby, are way cheaper than other things, like horses.
So hang in there!
Carolyn