I think ive read that plants can have their vegetative or flower buds get halfway or slightly damaged by cold which doesnt prevent the bud from develloping further but the damage may show later on. I remember an article about figs in particular, they can have their buds damaged by cold which results in deformed or stunted new growth in spring.
Maybe something similar is going on with your orange