Yeah, while I cant confirm or deny what Simon said about rootstock (if it is still potted I am not sure I agree with rootstock being the issue), that tree does not look healthy at all. I would start by removing all fruit. The tree is too small and not in a condition i would want to see it try to hold to maturity plus if it somehow ripened, it eill not be a quality or representative of the variety.
If you want to try and save the tree, either pot it into a bigger oit with a high quality pitting mix that contsins no added fertilizer (dont use a brand such as miracle grow, vigoro, etc.). It should not be a heavy/dense mix and the mix should be well draining. Oh, and no Black Kow or manure. Prune the tree back, somewhst heavily, and then leti it flush back out and tip prune to shape whrre needed. Make sure it get a good vonsistent watering (if you use the right mix it we ill dry out quite rapjdly) and fertilize lightly but on a regular basis. Make sure it has a nitrogen component for good fegetative growth (agan, I f growing in a pit, use a quality palm fertilizer).
If the tree is in the ground, follow the above leaving out the staps relating to growing in a pot.