No one can destroy you. All they can destroy is the friendship you shared between you, which must have been a flimsy friendship to begin with.