You were correct in realizing there was a point in time when you should have taken the initiative and requested to be removed from the case. Next time be proactive so that you have more control over your professional reputation. And get used to the idea of patients and their families complaining to the agency whether they have a justifiable reason or not. Some of them do it to manipulate you and the agency, others do it because it has become a part of their dysfunctional coping. It is best to find a second agency to work with, so that you always have the upper edge in having more than one client to keep you going when you are between cases. And don't let a bad situation go so this far next time. Don't feel bad, it is him, not you.