Your leaves are just old leaves being discarded by the tree. This is more or less a common experience this time of year. A citrus leaf has a life span of approximately 18 months. When a citrus leaf no longer can pull its weight, it is quickly discarded. You don't have anything to worry about.