If you clean your aligner with toothpaste, it scratches the surface and gradually turns it from clear to matt. A matt surface picks up more bacteria and colourings, which can make it turn yellow. If you clean your aligner with water only, that isn't enough, and saliva and bacteria stick to it. This shows up as a white or yellowish layer of plaque on the surface. It can be cleaned off, but it takes a lot of brushing and soap. If you wash it with soap and still notice a change in colour, try cleaning it more thoroughly. Other possible reasons for an aligner changing colour are: drinking coloured drinks with the aligner in; drinking coloured drinks without the aligner but putting it straight back in before the colourings have been rinsed away; and smoking with the aligner in.
This answer is for general information. Your own case needs an examination by an orthodontist.