I recently grit my teeth and turned down a potentially-lucrative assignment because they had not been timely in getting information to me. It would have paid well but, I feared that the unreliability would compromise my ability to generate results. And, they would have blamed me.
Two days later, something better came along and I'm immersed in that project.
Stick to your guns. Have the fortitude and the courage to turn something down. Something better will come. And, the clients you declined, will gain respect for you too.