I would remove the fruit once they are about marble size so that your young tree will put its energy into root and vegetative growth. You can stunt its growth if you allow it to hold fruit and with a minimal amount of leaves, the fruit quality may not be as good as it can be. If you just remove the blooms without allowing fruit to get to about marble size, your tree may bloom again, wasting more of its energy.
Simon