The ruling Social Democrats remained the biggest party with 40.6 percent of votes, marginally ahead of the centre-right Alliance, which garnered 40.3 percent in the polls, results showed after most votes were counted on Monday.
With more than four-fifths of ballots counted, Sweden's national election commission reported the governing Social Democrats had 28.1 percent of the vote, making it likely to lose a significant number of seats despite emerging with the most support.
But the traditional polls greatly underestimated support for the Sweden Democrats before the previous election, while the online surveys were far closer to the result.
The Sweden Democrats have promised to sink any government that refuses to give them a say in policy, particularly on immigration.
"Sweden needs breathing space, we need tight responsible immigration policies". Both the ruling center-left Social Democrats and the center-right Moderates had among their worst results in modern Swedish political history.
While most pollsters see the two blocs winning roughly equal support, the figures for the Sweden Democrats differ widely between surveys.
The results show Sunday's vote was one of the toughest challenges in decades to Sweden's social democracy, characterized by its high tax rates and substantial welfare system, aimed at reducing inequality through social inclusion.
Prime Minister Stefan Lofven, who brought the Social Democrats to power in 2014, said he meant to remain in the job.
Immigration was the hot topic of the campaign, helping the steady rise in popularity of the Sweden Democrats. He said his party had "won" the elections because of its gain in seats.
"I've said throughout the campaign that 20 to 30 percent (of votes) is a reasonable score for us and I think that's possible".More news: Botham Jean's killer confirmed as Amber Guyger
More news: SC declares emergency ahead of storm Florence
More news: CBS board discussing Moonves exit, a clearer way for a sale
The prime minister is usually the leader of the party with the most votes, but Sweden's fragmented political landscape after Sunday's election makes it impossible to guess who will form the next government. "And the Social Democrats and a Social Democratic-led government is a guarantee for not letting the Sweden Democrats extremist party, racist party, get any influence in the government".
But while there was a sense of relief among supporters of mainstream parties about the nationalist group's more limited than expected gains, the election underscored a broader shift to the right in one of Europe's most socially progressive nations.
Prime Minister Stefan Lofven used his post-election speech to again brand the Sweden Democrats as racist, pointing to its neo-Nazi roots.
"He said he was interested in cooperating with the other parties, and wanted to tell the Moderates in particular 'how to govern the country".
The Sweden Democrats want to freeze migration and have pushed for the country to leave the European Union. "We used to be a very calm nation", she said.
He said he planned to build a government that would "unite our country and take responsibility".
Lengthy negotiations will be needed to build a majority, or at least a minority that won't be toppled by the opposite side.
Lofven accused some of his opponents of being prepared to work with the far-right party, which he vowed his party would not to.
The opposition is intent on ousting Lofven, with some Moderates willing to go so far as to put an end to SD's pariah status and open negotiations with them.
The Swedish crown strengthened against the euro over the weekend, trading at 10.44 crowns at 0729 GMT against a last quote on Friday of 10.49 crowns, following the Sweden Democrats smaller than expected gains.