Mr Abiy said soldiers deployed to the contested town of Badme had experienced "psychological effects", according to the state-linked Fana Broadcasting Corporate.
"We should end this suffering, and fully return to peace," the prime minister is quoted as saying.
Ethiopia's previous leaders always said they accepted the 2002 ruling but they never actually implemented it.
Mr Abiy's announcement is especially significant as it comes after the release of thousands of jailed politicians, activists and protesters, including British citizen Andargachew Tsege who was being held on death row, and the promise of wider reforms.
What does Eritrea say?
Eritrea has not commented on Ethiopia's announcement but Information Minister Yemane Gebre Meskel had previously told the BBC that relations could not be resolved until Ethiopia withdrew "from the occupied territories".
"The ball is now in Eritrea's court," Tesfalem Araia from the BBC's Tigrinya service says.
"Eritrea has been on a war footing and the justification for forced conscription into the army has been the conflict with Ethiopia," he adds.
That forced conscription is the reason given by most of the thousands of Eritreans who flee the country, making the perilous journey to Europe.