Trip report!
I ended up getting a last min gig in San Jose and managed to get an extra day to explore. Taking everybody's advice I did the following:
First I hit two farmers markets, found some oddities (buddhas hand abounds) but nothing I hadn't had before.
Stopped at Tropical Fruits on Silver Creek Rd and found nothing, but the shop a couple doors down had an impressive collection: ambarella fruit, jackfruit, several guava varieties, dragon fruit, rambutan, etc etc etc.. all stuff I had, but some fruits I haven't seen outside SE asia. Stopped at nearby New Seasons Supermarket and picked up a Bacon Avocado.
Next I wandered around the cactus garden at Stanford and foraged a few odd cactus fruits.. gotta do some research to make sure they're edible.
I was about to turn back and go to the hotel at this point, but against my better judgement I took the 1+ hour trip through heavy traffic all the way up to Berkley Bowl. This was completely worth the trip, they had an impressive amount of fruit. I managed to find some Xoconostle, which I've never heard of before. These days finding just one new fruit makes a trip a success for me, so mission accomplished!
I'm going to come back again to wander South CA, using all the suggestions. thanks for all the help guys, this was a lot of fun!