He converted the spot-kick after Alan Hutton had handled in the area.
Kris Boyd had put Rangers in a winning position with a double but Kilmarnock refused to accept defeat.
Boyd scored at the end of each half and Frazer Wright contributed Kilmarnock's first goal.
Boyd looked sharp throughout and went close to breaking the deadlock in the 14th minute when he was through on Graeme Smith but pulled his shot wide.
Libor Sionko then tested the home goalkeeper but he proved up to the task to keep the scoreline blank.
Naismith then did well to head an effort from Julien Rodriguez off the line as Rangers pressed.
Kilmarnock had a strong penalty claim turned down when the ball struck Boyd's hand in the penalty area but referee Alan Freeland waved play on.
Lee Martin then forced another fine save from Smith after good work from Jeremy Clement.
But Kilmarnock also contributed to an engrossing contest and had another penalty claim turned down when Allan McGregor pulled Naismith.
Danny Invincibile also had a chance late in the first half but Hutton was across quickly to block his shot.
But it was Rangers who took the lead two minutes before the break when Boyd was given too much space in the box and he collected Dado Prso's pass before finding the net.
Allan Johnston forced a great save from McGregor at the start of the second half as Kilmarnock went in search of the equaliser.
Prso should have done better at the other end when he was set up by some excellent place by Sionko but the Croatian's shot was high and wide.
Smith then managed to get his hands to a fierce shot from Martin to send it wide for a corner.
But Wright equalised after 63 minutes with a superb header after McGregor failed to cut out a corner from Gary Wales.
Invincibile should have put the home side ahead shortly afterwards but headed tamely wide.
But Prso then fed Boyd five minutes from time and he tucked the ball between Smith's legs to put Rangers ahead but then came Naismith's late penalty drama.