Walking away from a problem contractor

My General contractor means well. he really does. But throughout the project he has been focused on other jobs and his subs have made every kind of error, from improperly leveling the floors, to improperly finishing the floors, etc. We’ve had to re-demo and re-do the master bedroom shower, the front door, the entire brownstone on the front steps, and more. But while all of that I am keeping a cool mind about, he just disappears and the last 10 items on my punch list will drag on forever. In fact, they’ve been dragging on for six months now because he ignores me, won’t call me back, says he will get back to me and doesn’t–you know the drill if you’ve done any contracting. I have no problem finding someone else to help me finish these jobs, and the money I owe him might just cover what’s left to be done. But given I do not have the final dept of buildings approval on the house, I wonder if anyone knows whether there are issues in walking away. Would I need a new GC to file for the final inspection? Is there a standard procedure for notifying him that I am cancelling the contract?