Bailey Peacock-Farrell saved from Haris Hajradonovic and Edin Visca hit the crossbar before Northern Ireland emerged from their first ever penalty shoot-out 4-3 winners thanks to Boyce’s strike.
It means Northern Ireland will have a chance to book a ticket to the rearranged Euro 2020 finals next month in Belfast when they face Slovakia, who saw off the Republic of Ireland in a shoot-out in Bratislava.
It was a dramatic finish to a tense night in Sarajevo, where Rade Krunic fired the Bosnians into an early lead before Niall McGinn levelled for the visitors eight minutes into the second half.
Bosnia were roared on by around 1,500 fans allowed into the Stadion Grbavica but victory would go to the visitors, a first win in charge for Ian Baraclough coming at the perfect moment and a glorious way for Steven Davis to mark his record-breaking 120th cap.
Bosnia named a makeshift back four in which former Nottingham Forest man Anel Ahmenhodzic made his debut, Branimir Cipetic picked up his second cap and Sinisa Sanicanin only his third, but Northern Ireland struggled to test them in the first half as the hosts dominated possession.
They were straight on to the front foot, with three corners and three free-kicks around the box piling pressure on Northern Ireland in the first 10 minutes alone.
But it was from open play that they took the lead with 13 minutes gone. Paddy McNair and Jamal Lewis had already slipped on a surface which Northern Ireland did not train on in the build-up to the match, and when Evans became the third player to lose his footing it proved costly.
As the Leicester man stumbled, Cipetic took his chance to pull the ball back for Krunic to fire in.
Baraclough’s side almost made an immediate response as Corry Evans’ long ball picked out Josh Magennis, but Ibrahim Sehic made a superb save to deny the Hull forward from close range.
But Bosnia still looked the more dangerous of the two sides, with Pjanic sending a shot whistling past the post as Peacock-Farrell could only watch before bending another chance narrowly over.
Edin Dzeko then pulled the ball back for Krunic to shoot straight at the goalkeeper.
Northern Ireland did end the half well though with McNair heading the ball back across goal for George Saville whose left-footed effort was deflected over.
And they continued on the front foot early in the second half, with McGinn sending in a dangerous cross from the left which almost found Corry Evans.
Northern Ireland started the second half on top and got their reward in the 53rd minute.
McGinn chased down a ball into the box, nicking it away from Cipetic, beating Sanicanin, and then firing past Sehic to level the match.
It could have been two moments later as McGinn sent in a glorious cross towards McNair but Sead Kolasinac did just enough to put him off right in front of goal.
Northern Ireland’s equaliser sparked an open period in the game as Saville and Lewis went close for Baraclough’s side while Peacock-Farrell saved from Krunic and Amir Hadziahmetovic while Pjanic hit the crossbar with a free-kick.
Gavin Whyte replaced Corry Evans with 17 minutes left and the Cardiff man almost put Northern Ireland in front when Sehic slipped and his snap shot was deflected wide, but the whistle went for offside.
Instead, Northern Ireland went into extra-time for the first time since 1958, starting it well as Jordan Jones brought a good save from Sehic before Peacock-Farrell got down sharply to deny Edin Visca.
But the two sides could not be separated until the shoot-out.