Gypsy, Roma and Traveller communities should be treated as voters and “not political footballs” during elections, campaigners have said.