I don't know about Socal or other locations, but here in South Florida, in over a decade of growing mango trees (about 3 dozen of them), I've never pruned fruit on young trees and it's never been an issue.
While pruning fruit is necessary for some species (eg, Jackfruit -- which will go into decline if allowed to carry too much fruit), mangoes will do one of two things: either they will abort the fruit on their own, or the fruit will turn out small. But neither case presents a detriment to the tree.
My sweet tart did the same thing actually. I planted mine out from a 7 gallon, and within a year, it produced 2 dozen fruits -- all of which I allowed to remain until maturity. The fruit were all excellent, albeit a bit small.