weigh upon one's conscience

• cause someone remorse, cause someone to feel regret