Heavy metals are governed by parent material of soils and influenced by the soil physicochemical properties and soil and crop management practices. This paper evaluates total heavy metal concentrations in rainfed soils under diverse management practices of tropical India. Vertisols (clayey soils with high shrink/swell capacity) had the highest concentrations of heavy metals. However, chromium (Cr) content was above the threshold value in Aridisol [calcium carbonate (CaCO3)]-containing soils of the arid environments with subsurface horizon development. Concentration increased at lower depths (>30 cm). Basaltic soils showed higher concentrations of nickel (Ni), copper (Cu) and manganese (Mn). Cadmium (Cd), cobalt (Co), Cu and Mn concentrations were higher in soils cultivated to cotton, whereas Cr concentration was above the threshold level of 110 mg kg−1 in food crop cultivated soils. As the specific soil surface is closely related to clay content and clay type, soil’s ability to retain heavy metals is more closely tied to the specific surface than to the soil cation exchange capacity. Higher positive correlations were found between heavy metal concentrations and clay content [Cd(r = 0.85; p ≤ 0.01); Co (r = 0.88; p ≤ 0.05); Ni (r = 0.87; p ≤ 0.01); Co (r = 0.81; p ≤ 0.05); Zn (r = 0.49; p ≤ 0.01); Cr (r = 0.80; p ≤ 0.05); Mn (r = 0.79; p ≤ 0.01)]. The amounts of nitrogen–phosphorus–potassium applied showed a positive correlation with Co and Ni (r = 0.62; p ≤ 0.05). As several soils used for growing food crops are high in Ni, Cr and Mn, the flow of these metals in soil–plant–livestock/human chain needs further attention.