I also think the tree is alternate bearing, having on and off years. You wrote that you fertilize the tree, however the leaves are showing a magnesium deficiency. Look at the analysis of the fertilizer you are using, to see if it contains magnesium (Mg).