The problem with the fruit drop is becouse of the leaf drop, no leaves means no food so no energy to sustain fruit.
The problem is probably in the soil, it can be too much moisture since i think this pot a little too large for the plant. Check if the soil at the drainage holes is wet or overly moist. Ive experienced citrus roots rot in a too wet of a soil. The other spectrum could be too dry, u can check for those two criteria.
It could be starved for food, thought my citrus are generally also way underfed, but i havent had a leaf drop becouse of this.