It came as John Whittingdale, the Culture Secretary, became the first Cabinet minister to break ranks and suggest that he could back the campaign to leave the EU.
Mr Cameron’s deal with Brussels was further undermined after official figures showed that his “emergency brake” on in-work will not affect most European migrants.
Meanwhile, investment bank Goldman Sachs warned that the pound could lose a fifth of its value if the UK votes to leave the EU.
There was growing anger among the Conservative rank and file on Thursday after Mr Cameron used a Commons appearance to tell MPs to “do what’s in your heart” and not listen to eurosceptic Tory associations.
Conservative members traditionally hold eurosceptic views and Downing Street is concerned that Tory associations will urge MPs to vote to leave the EU.
There were warnings last night that the comments could damage the party’s campaign in the local election and the London Mayor election, which is being fought by Zac Goldsmith.
“I was very saddened to read what the Prime Minister had said,” the board member said.
“He was talking about the same grassroots members who elected him as Leader of our Party, who pounded the streets delivering millions of leaflets and helped secure his victory in the last election.
“I have had activists contact me and they are very upset and concerned by the Prime Minister’s comments.
“It has been suggested that if we aren’t needed then activists won’t turn up and campaign for Zac as Mayor. If he doesn’t want our input then we won’t back Zac, or come out for the local elections this year.”
In a letter to The Telegraph Andrew Nicholas, the chairman of Enfield North Conservative Association, said: “Despite what he appears to think, he has no divine right to rule and if he thinks he can contemptuously dismiss the views of those of us throughout the Country who helped to put him in Downing Street, he may well find he is in for a nasty shock in the future.”
Councillor Osman Dervish, the chairman of the Romford Conservative Association, told the Telegraph he has already had members of his association threatening not to canvass for the Party.
And John Strafford, a former association chair who now runs the Conservative Campaign for Democracy, said: A dictatorship is when a leader says what he thinks and decides what is going to be done. I am afraid this is what David Cameron has done.
“There is huge frustration within the Party’s grassroots and there are other Party’s waiting on the fringes for bitter Conservative members at the end of this referendum.”