Most 3D reconstruction methods may only recover scene properties up to a global scale ambiguity. We present a novel approach to single view metrology that can recover absolute 3D heights of objects and camera parameters, namely, orientation, field of view and height above the ground, using just a monocular image acquired in unconstrained conditions. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights and the line at infinity, through estimation of their projections such as bounding boxes and the horizon. We leverage categorical priors for objects such as humans or cars that commonly occur in natural images, as references for camera calibration. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications like virtual object insertion. Further, the perceptual quality of our outputs are validated by a user study.