Conservative MP John Bercow has won the coveted position of Speaker of the House of Commons. But he may find his new role a poisoned chalice. Rumour has it that many prominent Tories are already working behind the scenes to unseat him. That old cliche that your fiercest opponents are on your own side is so true. The reason? We hate an apostate and Bercow's journey from hard-right Thatcherite to thoughtful social liberal with a Labour-supporting wife has sent some fellow Conservative MPs into apoplexy.

There's something about a former ideological soul mate performing a U-turn that riles us beyond description. Maybe because it challenges our own belief system and even, dare one say it, engenders a little bit of introspection about our own ideas. In short, we start to question our own foundations and this can be too much to bear for politicians who have built up an armoury of titanic self-belief. It applies outside Westminster as well. Right-wing commentators like Melanie Phillips – former Labour-supporting Guardian journalist – and Peter Hitchens – a former Trotskyite – attract raw hatred from former political bedfellows.

Politicians, in particular, hate turncoats with a passion. Take Michael Portillo. A former arch-Thatcherite – his famous 'Who Dares Wins' speech won him the support of the elderly Tory matrons and battling brigadiers at the Tory conference – but his move to the centre so antagonised his former colleagues that it effectively scuppered his leadership aspirations.

On the Labour side the defectors to the Social Democratic Party (SDP), Roy Jenkins, Shirley Williams and Bill Rodgers, were loathed for departing the Labour ranks. Jenkins was already an unpopular figure, openly derided by the Left even before he went to Brussels in the late 1970s. Although he was a miner's son, he simply sounded too posh for the Labour ranks. "The only thing Roy Jenkins fought for was a table for two at Le Gavroche" was a popular jibe from his own side at the time.

When Jenkins left for Brussels, Labour leftwinger Dennis Skinner delivered a memorable intervention. "I leave this party without rancour," said Jenkins in his farewell speech. Unfortunately, Jenkins' infamous inability to say his 'r's' made his pronouncement sound like something else entirely. So Skinner said, "You're not taking Marquand with you then?" a reference to David Marquand, another right-wing Labour MP who later joined the SDP.

When Jenkins died in 2003, one Labour MP was heard to mention "it's one down and two to go". Yes, unbelievable, but that's how much a defector is loathed. Also, in Labour ranks, the name Reg Prentice will go down as forever synonymous with betrayal. Prentice, a rightwing Labour MP, joined the Tories and became a minister under Margaret Thatcher. His own side never forgave him.

As for the Speaker of the House of Commons, Ann Widdecombe would have been my choice. When I worked at the Home Office in London, Ann – known affectionately as 'Doris Karloff' – used to sit alone in Politico's bookstore around the corner from Victoria Street at lunchtime having her soup.  A great lady who would have scared the hell out of the Commons and treated them rather like a headmistress chastising a bunch of wayward schoolchildren. Which is exactly what the House of Commons needed. Sadly, the Commons chose otherwise.