The prime minister has backed Jamaican foreign minister Kamina Johnson Smith to replace Labor as the public face of the 54-nation body.
But allies rejected Mr Johnson’s move and re-elected the Baron of Scotland for a second term in the Commonwealth Heads of Government (Chogm) Meeting on Friday.
UK officials, who have been discreet critics of the Baron of Scotland, were confident Mrs Johnson Smith would succeed.
“It’s a good day for democracy,” Johnson said at a news conference.
“I work well with the Baron of Scotland, it has been a very long time, since I became foreign secretary, and I look forward to working with her over the next few years.
“We will simultaneously do everything we can to help strengthen the Commonwealth secretariat and truly deliver value to the members of the Commonwealth.”
The Scottish baron said it was “very humble” to be reappointed in two years.
“It is a real honor and privilege to continue serving the family of our nations and I will do so to the best of my ability. We will face the world’s challenge with unity and purpose,” she added.
“Seeking high position is an act of profound service and I would like to commend my colleagues who have also sought to serve.
“The commonwealth is made richer by the breadth and depth of talented leaders who dedicate themselves to our family of nations.”
Mr Johnson has previously said his choice would have “broad experience and support to unite our unique family of nations and seize the opportunities ahead”.
But Mrs Johnson Smith admitted her defeat, writing in a tweet: “Always loved, I continue to serve and of course, hearty congratulations to the Baron of Scotland.”
Shadow Foreign Secretary David Lammy said: “This has been a divisive campaign and a humiliating diplomatic failure for Boris Johnson, a testament to his shameful lack of influence within the Commonwealth of .”