A transition from a smooth torus to a chaotic attractor in quasiperiodically forced dissipative systems may occur after a finite number of torus-doubling bifurcations. In this paper we investigate the underlying bifurcational mechanism, which is responsible for the termination of the torus-doubling cascades on the routes to chaos in invertible maps under external quasiperiodic forcing. We consider the structure in the vicinity of a smooth attracting invariant curve (torus) in the quasiperiodically forced Hénon map and characterize it in terms of Lyapunov vectors, which determine the directions of contraction for an element of phase space in a vicinity of the torus. When the dependence of the Lyapunov vectors upon the angle variable on the torus is smooth, regular torus-doubling bifurcation takes place. On the other hand, we observe a transition consisting of the appearance of a nonsmooth dependence of the Lyapunov vectors upon the angle variable on the torus. We show that torus doubling becomes impossible after this transition has occurred, although the attractor of the system still remains a smooth torus. We illustrate how the transition terminates the torus-doubling bifurcation line in the parameter space with the torus transforming directly into a strange nonchaotic attractor. We argue that the transition plays a key role in mechanisms of the onset of chaos in quasiperiodically forced invertible dynamical systems.