We demonstrate that nonstoichiometry is one of the main reason of strong deviation of the InN optical gap in the 0.7-2 eV range, with N/In 〈1 and N/In〉 1 corresponding to the lower and higher energies, respectively. The phenomenon is discussed in terms of atomic orbital energies, which are strongly different for indium and nitrogen, therefore both excess atom incorporation and elimination could change the optical gap. We estimate such trends using the approximation of the empirical nearest-neighbor tight binding theory. It is also demonstrated that resonant absorption in In-enriched regions is an additional factor lowering an effective absorption edge.