The Japanese government, which began scientific whaling in 1987, a year after the global whaling moratorium was introduced, has long maintained that most whale species are not endangered - six of the world's 13 "great whale species" are classified as endangered, according to the World Wildlife Fund.
The research programme was criticised as being a cover for commercial hunting.
"Regrettably, we have reached a decision that it is impossible in the IWC to seek the coexistence of states with different views".
Japan is now conducting research whaling in the Northwestern Pacific and the Antarctic Ocean, but it must halt such activities in those waters once it withdraws from the IWC.
It was not yet clear how many whales would be caught each year once Japan resumes commercial whaling, officials said.
Japan has hunted whales for centuries, and the meat was a key source of protein in the immediate post-World War II years when the country was desperately poor.
Tokyo maintains most whale species are not endangered, and eating whale meat is part of its culture.
The decision, some experts said, allows Japan to save the money it spends to support Antarctic whaling while taking a tough pro-whaling stance - a matter of national pride for some conservatives.More news: Browns' Baker Mayfield denies staring down Bengals' Hue Jackson despite video evidence
More news: Pakistan court sentences ex-PM Nawaz Sharif to 7 years in jail
More news: Bears Twitter Trolls 49ers After Dirty Hit on Mitchell Trubisky
"The Government of Japan must urgently act to conserve marine ecosystems, rather than resume commercial whaling", Annesley said.
Nations against whaling, led by Australia, the USA, and the European Union, blocked Japan's proposal by 41 to 27 votes, the Japan Times reported. "Japan will be pressed to undertake a fundamental reassessment of its position as a member of the IWC", he said.
Ahead of the announcement, activist groups warned Japan against the withdrawal.
Greenpeace Japan Executive Director Sam Annesley accused Japan's government of "trying to sneak in this announcement at the end of year" to avoid worldwide scrutiny.
The IWC banned commercial whaling in 1986, but some countries including Japan, Norway and Iceland have exploited a provision in the 1946 Convention for the Regulation of Whaling that allows whales to be killed for scientific purposes.
Japan has defied the protests to conduct what it calls scientific research whaling, having repeatedly said its ultimate goal was to whale commercially again.
It makes no secret however of the fact that meat from the expeditions ends up on dinner tables.
"It is a major decision that will protect domestic fishermen who live on whaling", Kazutaka Sangen, mayor of Taiji, Wakayama Prefecture, said about the decision.
"At the IWC general meeting in September this year, it became evident once again that those supporting the sustainable use of whale stocks and those supporting protection can not co-exist, leading us to this conclusion".