I really like perlite, I also use my own worm compost, and I also use really good, peat-based potting soil (already has bark, guano, and other nutritious stuff in it).
I usually use 5 parts potting soil, 2 parts perlite (really cheap for me, we have a perlite mine here in Idaho), and 1 part worm compost. Lately I have used a suggestion from Millett and mixed in some Turface MVP. I like that stuff!
Because I grow everything in pots, a mucky bottom layer in my pots is usually a death knell for my trees. So the big thing is drainage, drainage, and more drainage.
Everything seems really happy, and a lot of my trees over over a decade old, produce fruit, and seem like they are enioying life.
Carolyn