Yeah, probably best to let it set fruit and then cut them off. Lychee flowering is heavily influenced by average nightly low temperatures. I am unfamiliar with the climate in your area but if you prune off flower panicles on Lychees when it is still cold enough, the tree may re flower and expend even more energy on secondary blooms or mixed blooms.
Simon