some leaves have started to fall off
This can happen for a few reasons. This needs to be planted in a very shady spot. Too much sun will cause yellowing leaves.
Soil that is not acidic enough will lock out nutrients. This can, also, lead to yellowing leaves.
Make sure that the plant is not buried too deep or too shallow. This can cause problems as well.
Here are some articles that will help:
https://www.gardeningknowhow.com/ornamental/shrubs/skimmia/growing-japanese-skimmia.htm
https://www.gardeningknowhow.com/garden-how-to/soil-fertilizers/testing-soil.htm
https://www.gardeningknowhow.com/garden-how-to/soil-fertilizers/soil-ph-plants.htm
The white blossoms have a delightful floral scent in the Spring when in full bloom.
My husband moved my Skimnia into full Greek summer sun and burnt the plant, I moved it under a tree the following summer and watered it regularly, but it was eaten by bugs and grass hoppers - the plant looks very sorry for itself, can I cut it right back in the hope it recovers or do I give up, but another and not let hubby anywhere near it? Many Thanks
You can prune it back, some. By about 1/3 will be fine. Do this during winter dormancy, or before breaking of spring growth, however. This should not be done during the active growing season. Proper care, next year, will help. Be sure that the soil's growth parameters are within range, and slightly acidic.
https://www.gardeningknowhow.com/ornamental/shrubs/skimmia/growing-japanese-skimmia.htm
https://www.gardeningknowhow.com/garden-how-to/soil-fertilizers/testing-soil.htm
https://www.gardeningknowhow.com/garden-how-to/soil-fertilizers/soil-ph-plants.htm
my plant are in a pot, and can it stay outside in the winter how much frost can it takes ?
They are hardy down to USDA zone 6. This means that they can withstand temperatures as low as about 5F (-15C).
https://www.gardeningknowhow.com/ornamental/shrubs/skimmia/growing-japanese-skimmia.htm