Actually, I am surprised more people don’t grow lychee trees in SoCal. I see many Loquat, Avocado and Citrus trees in my area but no Lychee Trees. They produce delicious fruit and are very attractive specimens. Lychee Trees require above average watering, sprinklers by themselfs are inadequate, but at my location are disease free. Just plant them in good sandy soil and water very frequently especially in the spring and summer. If you plant young trees in heavy clay soil you are likely to have slower growth, perhaps much slower, as the roots have difficulty penetrating to dense clay partials. A old timer who had been growing lychee trees for many years once told he it takes 6-7 years after planting in the ground for a lychee tree to begin to getting consistent fruit production.
Over the last four years my trees have grown well here in HB. Enclosed is a photo from July 2020 when I planted my young Brewster lychee tree from a standard #15 pot.
Johnny

Brewster Lychee Tree just Planted (7-12-2020)