Longan is definitely easier to grow than lychee. It's more drought tolerant and has a little bit more cold tolerance. Longan also fruits more readily.
Lychee trees can be very finicky in Southern California, and need to be consistently and adequately watered, especially during the Summer. They don't like it too dry, and have problems with wind also. Lychee trees are slow growing, especially very small trees, often a bit frustratingly so. They'll steadily and gradually grow but it will be at a slow rate.
I like lychees much more than longan though, and if you can get it to fruit and grow, the payoff is well worth it.