When fruits reach their mature size, squeeze the fruit with your fingers. It has to be very firm, but slight give in is what you should be looking for. Pick the fruit that is still green, keep it in the brown bag until it turns completely black/brown. The softer the better. Then it's ready.
You can let the fruit brown on the tree, but the animals can get it or if you miss it, it will drop to the ground and will get ruined as it becomes very tender.