None of these groups who are supporting Brown are terribly enthusiastic about his record or his positions on issues. Many of them would support Kennedy in a second if he was running as a Republican challenger to Brown and had a chance of getting elected. Some of the more socially conservative groups have no candidate they like in the race at all. Yet all of these Tea Party associated groups have abandoned the candidate with genuine Tea Party credentials to support Brown because it is a better political strategy. They may not like Brown much, but if he can win a Senate seat from Massachusetts that's such a huge blow to the Democrats and would do so much to weaken the efforts to pass Obamacare that they are biting the bullet and promote a candidate who they wouldn't give a second look in another circumstance.
In return, Brown has really embraced the Tea Party label and has borrowed their message and many of their ideas in pushing his candidacy. This raises the hope for those who are reluctantly supporting him that some of that rhetoric will stick with him once he gets in office and he will be a better representative for the people than he would have been without their support, either by being educated from his association with the Tea Party movement or out of a sense that he owes something to them for putting him in office.
All of this seems pretty unfair for Joe Kennedy who really is a good candidate with solid credentials and interesting ideas and a more authentic Tea Party platform. Some of his supporters are taking it personally. There is already a lot of resentment among more libertarian Tea Party activists against some of the more mainstream and often better-funded groups which have become involved. They are seen as corporate shills or interlopers from the Republican party or opportunists trying to cash in on Tea Party momentum. There is fear that they will take over the movement and resentment that they give the left-leaning media a basis to criticize the movement as a whole as illegitimate.
The groups which have stuck with Kennedy are the ones which are most ideologically driven and which put ideals and principles ahead of political pragmatism. But this sort of misses the whole point of the Tea Party movement, which is to actually influence government and implement changes in policy. You can't change anything with candidates who can't get elected, no matter how great they are. It's the old, old argument of whether or not to take the lesser of two evils, and one element of the Tea Party movement has decided that evil is still evil and utterly unacceptable while the rest have taken the position that less evil is better than more evil.