Certified GKH Gardening Expert
Unfortunately, there is much conflicting information about this subject. It would seem that if they are not used to cold then they don't handle freezing temperatures very well, but it they are occasionally exposed to cool temps while they are younger then they tend to tolerate it a little more.
Some cultivars can handle below freezing down into the upper twenties, while others can barely handle frost.
This article will give you information on caring for this plant: https://www.gardeningknowhow.com/ornamental/shrubs/osmanthus/tea-olive-cultivation.htm