I feel your pain!
I also get super low fruit set. I have noticed that I get at most 1-2 fruit from a few grafted trees I got here in PR and 0 from the trees from Florida. Only Julie gives me more than a handful of fruit each year. But the wild trees that grow from seed are full of fruit.
All the trees "grow well", I get lots of leaves. The PR "born" trees flower, only some of the Florida "born" trees will set flower, but no fruit.
I bet it is a similar problem as Happyisland. Would also know how to fix it!