I probably should treat this like growing mango in SoCal: "Plant a seed, then graft after it's ready to bear fruits", or else I would be dealing with this bloom every year, holding up the growth. (Idea from Simon). I now realize that it's not a good idea to graft this little seedling that early.
So, I tend to remove the graft (only set me back 2 months), let the rootstock grow bigger, then regraft something like Reed and Pinkerton combo on it.
Yes, it looks darn close to the fence from the photo, but it's got about 3 feet distance, and I'll probably change the cultivar to Reed/Pinkerton combo instead...
Thanks for your response, Brad!