I have a mature Sweetheart in the San Gabriel Valley area. I have had it in the ground for about 11 years. It is now about 7-8 ft tall and about that size wide with a large drooping canopy. No matter how to try to shape it, the new growth keeps drooping to the ground especially when they hold fruits. The first 6-7 years, I would get inconsistent blooming and may be lucky to get one fruit. The last 2-3 years, the blooming is more regular. Right now, there are many panicles of flower buds, even with the dry winter that we have had. Last year, I had a good harvest. I think the cool spring in May and early June 2017 helped the fruitlets to hold. Usually, the heat of late May to early July just beat up on the fruitlets and they would drop like flies. As you know by now, growing lychees in Southern California is a challenge. If it would be easy, you would have seen locally grown lychees in the markets some time ago.
Your tree being 4 ft tall is just too young. I would just enjoy it growing and give it time. If you are like me, you would get rewarded in due time. Surprisingly, my lychee tree has grown quite well. The major wind storm 6 years ago caused some large branches to break. But it has recovered well since then. The tree branches are quite weak. A heavy sustained wind can easily snap a branch or two. Good luck.