I have a very small garden pond. It has been covered in duckweed for several years. Manual removal never suffices. I dislike using chemical weed killers of any sort, but if you can suggest one that is not too generally harmful I would risk it. There is one lone goldfish too, the last survivor.
Try tying dry straw in a bundle, about the same size as a small bunch of flowers, and float it on top. Eventually it will start to sink and may help to solve the problem without murdering Goldy.