A cricket club has written to Prime Minister Sir Keir Starmer pointing out they are "not the baddies" after he referenced the club in a policy speech.