It depends on your feelings for the person. If by the time you break up, the two of you have already drifted apart, then I think it would simply be closing the door on something that didn't fit. But, if one or the other of you had deeper, more committed feelings, then the breakup could be devastating.