Kesar is a slow-growing mango tree even in Inda where it is native.
This is a poor choice for SoCal with our marginal climate. Even removing all of the fruit growth will still be poor.
Best to start over and select vigorous varieties with a higher probability of success. Jakarta, Sunrise, and 0-15 are all much better choices for our area with the Indian Flavor profile.
As to your small Kesar tree, I would just enjoy the fruit you get every year and plant some Mango seedlings (Manila, Kent exct) now and let them grow so you can graft some more desirable cultivars in a few years.
Johnny