We present a theoretical analysis of the optical light curves (LCs) for short-period high-mass transiting extrasolar planet systems. Our method considers the primary transit, the secondary eclipse, and the overall phase shape of the LC between the occultations. Phase variations arise from (i) reflected and thermally emitted light by the planet, (ii) the ellipsoidal shape of the star due to the gravitational pull of the planet, and (iii) the Doppler shift of the stellar light as the star orbits the center of mass of the system. Our full model of the out-of-eclipse variations contains information about the planetary mass, orbital eccentricity, the orientation of periastron and the planet's albedo. For a range of hypothetical systems we demonstrate that the ellipsoidal variations (ii.) can be large enough to be distinguished from the remaining components and that this effect can be used to constrain the planet's mass. As an example we presend KOI-13b (candidate exoplanet system) included in the September 2011 Kepler data release. The Kepler light curve shows both primary and secondary eclipses, as well as significant out-of-eclipse light curve variations. We model the relative contributions from (i) thermal emission from the companion, (ii) planetary reflected light, (iii) doppler beaming, and (iv) ellipsoidal variations in the host-star arising from the tidal distortion of the host star by its companion. Our analysis, based on the light curve alone, enables us to constrain the mass of the KOI-13.01 companion to be MC = 8.3 ± 1.25 MJ and thus demonstrates that the transiting companion is a planet. The teqnique is useful for current and future space missions such as Kepler and PLATO.