More is better, but the biggest impact is from a well aerated growing medium. Plants with sensitive roots can be tough to grow in pots; I've had bad luck with mangoes, for example (although I know it's faired better for other people). Plants that love water and organic-rich soil however thrive in them. The downsides to side holes are A) they can sometimes lead to water escaping when you water your plants, depending on the details of your soil, and B) at least for indoor cultivation they can be another entry point to the soil for pests like fungus gnats. But they do help aerate.
Might want to consider a copper hydroxide coating if these pots are going to be long-term homes for your plants. It helps prevent root binding by stopping roots from growing when they get to the walls rather than wrapping around them.
Some people are advocates of plant bags rather than plant pots. I've personally never tried them. They're designed to aerate well inherently.
Your ability to get fruit will depend on A) pot space, and B) light. Don't shortchange either. I'm rather light-limited here (only under grow lights, and not enough), so it mostly limits me to berries and other small fruits (although some plants still get quite sizeable), and my yields are low (my acerola yields are measured in weeks per fruit rather than fruits per week
). Shortchanging pot size can be just as bad. If you want, say, a banana to fruit and have a full yield, its pot needs to be the size of a rain basin. The more you shrink it, the smaller the yields will be, until it's too small to bloom or even reach full size.
I don't grow avocado and have only one citrus (which I'm only cultivating as a rootstock until I can get a good scion up here), so I wouldn't be the one to ask.