I see a lot of carrotwood seedlings around that looks similar. Since you are in SoCal, there a lot of them in parkways of houses. I have one pretty large tree in my front and it has a fruit that is a hard shell with orange seeds. They seem to grow easily everywhere it drops, it may be considered an invasive plant. The birds may be picking the fruits and dropping them around since I also see a lot of them in my back yard where there is no tree.