Brackets bond best to tooth enamel, but they also hold securely on other surfaces such as fillings or composite build-ups. To create a strong bond between the two materials, the surface of the build-up has to be roughened (lightly abraded with the dental drill). Once the bracket is removed from that tooth, the composite can be polished with no lasting damage. Bear in mind that bonding is sometimes done to change how teeth look in position (to make them appear straighter). If you decide to straighten your teeth after this kind of work on your smile, the bonding will probably need to be removed so that the braces can straighten the crooked teeth.
This answer is for general information. Your own case needs an examination by an orthodontist.