I've seen ants on sapodillas when there were no harmful insects like aphids. I think they like split fruits and suck on their sweet juices. Here also the ants like to nest on sapodilla trees. They get under the lichens, moss, and bark of the trees. They're not always doing damage, and as Adam pointed out previously, they can often do some good. Anyway it's good to be observant either way and not too rapidly jump to conclusions.