In a vote of Liberal Party members, Mr Sharma defeated Andrew Constance 251-206 in the final ballot on Sunday.
A former state treasurer and member for Bega, Mr Constance had been the firm favourite after being endorsed by the Opposition Leader Peter Dutton.
However moderates within the party backed Mr Sharma, the former federal member for Wentworth who lost his seat to independent Allegra Spender in the 2022 election.
Mr Sharma said he has a good relationship with Mr Dutton, despite not being his top pick.
“I have always had when I was in parliament, and since I’ve left as well,” he told ABC’s RN.
“I’ve got absolutely no problems working with him and with other members of the Liberal parliamentary team.”
Mr Sharma said he didn’t ask for Mr Dutton’s backing when he put himself forward.
Australia’s former ambassador to Israel said taking over from former senator Payne was a privilege.
“I would like to thank the party members for the opportunity to hold the Albanese government to account in the Senate over its many missteps and wrong decisions, and to fight for the many households across NSW struggling to deal with Labor’s cost of living crisis,” Mr Sharma said.
“The opportunity to serve in the Senate will allow me to fight for our nation’s national security interests in a time of greater global turmoil.”
Marise Payne announced her retirement from the Senate in September. (Mick Tsikas/AAP PHOTOS)
Mr Dutton said Mr Sharma’s entry to the Senate would come at a crucial time.
“His diplomatic and foreign policy expertise will lend considerable weight and wisdom to the public policy debate given the precarious circumstances in Eastern Europe, the Middle East and the Indo-Pacific,” the opposition leader said in a statement.
Deputy Opposition Leader Sussan Ley called Mr Sharma a fitting replacement for Ms Payne.
“Over the past 20 years, Dave has sat in the Oval Office with American presidents, helped to broker international peace agreements and has first-hand experience on-the-ground in Israel as a former ambassador,” Ms Ley said.
She said Mr Sharma would be a great addition to the Liberal team as it worked to win back federal seats lost to independents.
“Dave is someone who very much appeals to that cohort of voters which we need to win in 2025,” she said.
Australia/Israel & Jewish Affairs executive director Colin Rubenstein said Mr Sharma “has shown himself to have moral clarity and courage, including on important issues relating to the Australian Jewish community and Israel”.
Also passed over in Sunday’s vote was former ACT senator Zed Seselja, a staunch conservative who hails from the right faction.
Mr Seselja lost his Senate spot to independent David Pocock at the last election, leaving the ACT without Liberal representation for the first time since the territory started with two senators in 1975.
Monica Tudehope had also been touted as a strong contender and had the backing of Business Council chief executive Bran Black.
Ms Payne announced her retirement from the Senate in September after 26 years in parliament.