Cooperative control of functional electrical stimulation (FES) and electric motors in a hybrid exoskeleton may benefit from fatigue measurements and online model learning. Recent model-based cooperative control approaches rely on time-consuming offline system identification of a complex musculoskeletal system. Further, they may lack the ability to include measurements from muscle sensors that monitor the FESinduced muscle fatigue, which may hinder maintaining desired muscle fatigue levels. This paper develops an online adaptive reinforcement learning approach to control knee extension via an electric motor and FES. An optimal tracking control problem that uses an actor-critic identifier structure is formulated to approximate an optimal solution to the Hamiltonian-Jacobi- Bellman equation. The continuous controller provides asymmetrically saturated optimal control inputs of FES and the electric motor. Critic and identifier neural networks are designed to simultaneously estimate the reward function and the system dynamics based on sampled fatigue measurements and compute control actions. Importantly, simulation results show that a satisfactory joint angle tracking and actuator allocation can be obtained at multiple on-demand desired muscle fatigue levels and prolong FES utilization.