I didn't exactly put mine in a prime spot, one of the lowest spots in my yard and it's doing fine. Although, I do agree that it developed a lot of flowers, but only a few fruit. I mainly contributed this to the fact that I received the tree in very poor condition and it's only been in the ground a few months.
I did put some Black Kow manure around the tree and lightly fertilized it. That may make the difference here.