The C. hystrix is cold sensitive, and Australia is just coming out of winter, Melbourne would be having cold frosty mornings and nice warm sunny days later.
The plant is in a pot, so it might be in part due to inactive roots in cold soil, but warm leaves. It seems that the older leaves are affected but new leaves are clean ?
Kaffir Lime seedlings are fussy, and grafted plants can grow well, but still be fussy on some rootstocks.
Maybe a liquid seaweed fertilizer might help.
A copper spray might be protective.
Another alternative is OCP Eco Carb fungicide based on Potassium Bicarbonate, or Eco Carb Plus, based on Potassium Bicarbonate + Potassium Silicate.
Unless you have a confirmed pest or disease problem, I wouldn't remove leaves just for looks as you will stunt the growth of a young plant.