Yes. In fact, Alex (Squam) during one of his latest presentations (which can be found on another thread), said that irrigating mango trees once or twice per week during the dry season helps prevent fruit abscission.
And if we start talking about less drought tolerant trees (eg, jackfruit, soursop, jaboticaba, etc), harvest can be substantially affected when in drought stress -- both in terms of fruit size and amount.
I tried it both ways. For years, I didn't irrigate at all. Mango crops were smaller, and some trees didn't really want to fruit at all (jaboticaba, soursop, etc). With irrigation, crops are quite a bit larger, and trees are visibly healthier during the dry season.
The other thing to keep in mind is that water is what transports nutrients from the roots to the leaves. So nutrient absorption is slowed during periods of drought. And if this happens to correspond to the period when fruits are developing, that means less nutrients available to developing fruits (probably less brix as well).
Also, keeping soil moist is key to keeping soil microbes, worms, etc happy. You'll notice that when the soil is dry, worm castings will disappear.
The idea of zero-input gardening is cute. But providing an environment in which your trees can thrive is key to a great harvest.
I could see this for 3 gallon size... But, do you or people like Squam with large trees actually continue to irrigate twice a week when the trees are large like his???