I have visited SD botanical garden several times and had no issues with sampling fruits from the heavy producing trees (pitanga, rose apple, stone fruit). I wouldnt recommend cutting down banana racks or harvesting their mangos. That might draw their ire.
Exotica is a great place to visit, sample fruit, and check out all the rare plants. If I'm in the area, I sometimes drop by just to walk through and enjoy the oddities. There should be plenty of fruit in season over there.
Those are the only two places I know of were you can find some tasty treats.