Leaves, never.
Branches, however, within 12 inches of the ground are pointless when young. Cut them off. As it matures, you can remove branches even higher due to the branches weeping.
If they're touching the ground, then you invite more access points for pests and other issues as well.
You can, however, shorten those branches into thickening and making some lower fruiting wood. This can work if you have good lighting position, but not as effective at maturity.