I'm not an expert, but it looks like the pot you are using might be one of those types of pots that don't drain so well. Citrus needs a lot of water outside and in the heat (especially during flowering and to hold/grow fruit), but it also hates to be soggy wet. Is the bottom of the pot draining well, or soggy? You might be getting pretty root-bound in the pot too, which appears to be quite small for a three year old tree.