The former federal minister, who had been in poor health for some time, died in a Delhi hospital on Thursday night.
His political career, which spanned more than half a century, saw many ups and downs in recent years.
But until the end, he maintained a cordial relationship with leaders from across the political spectrum.
At the time of his death, Yadav was part of the Rashtriya Janata Dal (RJD) after he merged his own party, the Loktantrik Janata Dal (LJD), with the larger political organisation last year.
RJD leader Lalu Prasad Yadav, who was a friend who became a rival and then again an ally, said on Thursday night that the differences between them had never led to any bitterness.
"We shouldn't have said goodbye like this," said Mr Yadav, who posted an emotional video message from Singapore, where he has been recuperating after a kidney transplant.
Prime Minister Narendra Modi, who belongs to the Bharatiya Janata Party, said on Twitter that he was "pained" by the news.
"In his long years in public life, he distinguished himself as an MP and minister," Mr Modi wrote, adding that he would always "cherish" their "interactions".
Congress leader Rahul Gandhi visited Yadav's family members in Delhi to offer his condolences.
"I have learnt a lot about politics from Sharad Yadav ji. His passing away today has left me saddened. He shared a relationship of respect with my grandmother [Indira Gandhi]," Mr Gandhi told reporters on Friday.
Yadav, who was born in 1947 in Madhya Pradesh state, studied engineering before he chose to enter politics.
He was among a generation of politicians who were influenced by towering socialist leader Ram Manohar Lohia and who began their careers by participating in mass movements against then prime minister Indira Gandhi's government.
Yadav won his first election in 1974 with a thumping majority against the then-behemoth Congress party.
In the late 1980s, he played a big role in ensuring the implementation of a federal government commission's recommendations to extend affirmative action programmes to Other Backward Classes, which refers to a swathe of lower and intermediate castes in India.
He was a minister in federal governments led by VP Singh and Atal Bihari Vajpayee.
In 2003, he was among the leaders who founded the Janata Dal (United) - he left the party in 2018 after 15 eventful years. He then formed the LJD, which did not make much of a political impact.
On Thursday night, his daughter announced his death on Twitter with three words in Hindi: Papa Nahi Rahe (Father is no more).