As a powerful tool for medical diagnosis and bioanalysis, conventional optical spectrometers are generally expensive, bulky and always require an accompanying data processing device. In this work, we developed a novel smartphone-based CD-spectrometer (SCDS) for high sensitive and ultra-portable colorimetric analysis, with the advantage of cost-effective and simplicity. The distance between the light source and slit, the structure of SCDS and the parameters of camera in the smartphone were all optimized to ensure the best analytical performance. Besides, the SCDS employed HSV color model and utilized the overall intensity calculated by summing V-value of adjacent position for the absorbance measurement. In this way the errors caused by the low resolution of CD-grating can effectively be eliminated to promote the sensitivity of the SCDS. The performance of the SCDS was first validated for colorimetric detection of BSA with a detection limit of 0.0073 mg/mL, which is superior compared to that of the microtiter plate reader (MTPR). Moreover, by combining with 3,3′,5,5′-tetramethylbenzidine-manganese dioxide (TMB-MnO₂) nanosheets reaction, a high sensitive and specific system for ascorbic acid detection was established. The SCDS gives a detection range from 0.6250 μM to 40 μM with a detection limit of 0.4946 μM for AA detection. Compared to other studies, the SCDS features wide detection range and very low detection limit with low cost instrument. Therefore, the SCDS will be an ideal and promising colorimetric system for point-of-care (POC) application in food security, disease diagnosis and environmental monitoring.