David Starkey has been criticised for saying slavery was not genocide because of the survival of "so many damn blacks".
The TV historian and author made the remarks on an online show hosted by the conservative commentator Darren Grimes.
Writing on Twitter, former chancellor Sajid Javid said: "David Starkey's racist comments are a reminder of the appalling views that still exist."
Starkey has not yet responded to the BBC's request for comment.
Fitzwilliam College, part of Cambridge University, where Starkey teaches, described his remarks as "indefensible".
"We support and promote freedom of speech in our academic community, but we have zero tolerance of racism," the college said in a statement on Thursday. "Dr David Starkey's recent comments on slavery are indefensible."
They added: "The matter of Dr Starkey's Honorary Fellowship will be considered by the Governing Body at its meeting next Wednesday."
Starkey made the offensive remarks in an episode of Grimes's YouTube show Reasoned, entitled "Dr David Starkey: Black Lives Matter Aims To Delegitimate British History".
In it, Starkey said: "Slavery was not genocide otherwise there wouldn't be so many damn blacks in Africa or Britain would there? An awful lot of them survived."
He also claimed that the Black Lives Matter protests, following the death of George Floyd, had been characterised by "violence" and "victimhood".
He described cancel culture and the pulling down of statues as "deranged".
Image copyright Reuters Image caption Black Lives Matter protesters in Hyde Park in JuneThe academic went on to discuss the links between slavery and the British Empire.
Starkey said: "As for the idea that slavery is this kind of terrible disease that dare not speak its name, it only dare not speak its name, Darren, because we settled it nearly 200 years ago."
"We don't normally go on about the fact that Roman Catholics once upon a time didn't have the vote and weren't allowed to have their own churches because we had Catholic emancipation."
Nicholas Guyatt, a lecturer at the University of Cambridge, tweeted: "Can't speak for my employer but as someone who teaches history at Cambridge I'm ashamed of our connections with David Starkey and urge both the University and Fitzwilliam College to cut all ties with him."
It's not the first time Starkey and writer has been involved in a public race row.
In 2011, the BBC received nearly 700 complaints about Starkey's claim that "whites have become black", during a Newsnight discussion about riots in the UK.
Follow us on Facebook, or on Twitter @BBCNewsEnts. If you have a story suggestion email entertainment.news@bbc.co.uk.