In the Pacific Northwest, I've found it's not good to plant them out in the ground until late May, despite weeks of warm temperatures that may begin much earlier in the year. I've tried transitioning out multiple hardy citrus plants into the ground in mid-March to April and it never ends well, the leaves turn yellowish after a few weeks and do not recover, even sometimes some die-back (despite temperatures always being well above freezing). I'm not sure whether the temperatures just get too cool for the plants to do well, or whether they have difficulty with the sudden transition from going inside a warm grow area to outside that early in the season.
With hardy gardenias on the other hand, I did plant them out in January, before the snow came, and they did just fine, even kept all their leaves. I planted them as soon as they arrived in a box from the mail order nursery. I don't know in what conditions the nursery had kept them. I had a different hardy gardenia that had been growing inside under warm grow conditions and then I moved it out during a warm week in the middle of March and it did not do so well, had a lot of die-back and leaf loss. So maybe it is the temperature transition that is harmful. Or it could have been that this variety was inherently less hardy than the variety of the other ones, so hard to say.
I suspect that as the plant grows, the plant tissue becomes adapted to the temperature conditions it grew in, and that it takes more than a few weeks for the plant tissue to be able to adjust to lower temperatures.