This study provides the first inventory of small non-volant mammals in the Parque Estadual das Fontes do Ipiranga (PEFI), a protected area in the city of Sao Paulo, Brazil. The data was collected from 2015 to 2017 in 16 research campaigns with a duration of five days each. Four areas with different phytophysiognomies were sampled throughout the dry and rainy seasons. We sampled small mammals using live-capture and pitfall traps. Eleven species were captured, composed of six rodents and five marsupials. The sampling effort involved 5,600 traps/night, there were 527 capture events, and we captured 302 distinct individuals: 174 marsupials and 128 rodents. Recaptures accounted for 42.7% of the total captures. No significant differences were observed in the richness and abundance of small non-volant mammals between the different phytophysiognomies. We also found no significant differences in the richness and abundance of small non-volant mammals between the dry and rainy seasons. The relative abundance (Ar) and constancy index (C) of the species showed that the three most abundant and common species in the PEFI are: Didelphis aurita, Akodon montensis and Oligoryzomys nigripes, which represented 93.7% of the captures. Tomahawk traps accounted for 69% of the total captures, and pitfall traps were responsible for detecting the greatest richness, capturing 81.8% of the species. Comparing the efficiency of the different sampling methods in capturing small mammals in the PEFI, we observed significant differences between both pitfall versus Tomahawk and pitfall versus Sherman in the understory. The results obtained in this study are consistent with the past and current situations of the forest fragment which are in recovery after significant altered by anthropic activity. In light of this scenario of degradation and isolation, a defaunation process affecting the mastofauna is very likely in the PEFI, which favors the establishment and dominance of generalist species. This study could be the basis for further monitoring programs of small non-volant mammals. The data obtained here will also increase knowledge about the diversity of small mammals in urban fragments of the Atlantic Forest and demonstrate the importance of the PEFI for the maintenance of ecologically important species within the largest metropolitan region in Brazil. These species play important biological roles for the maintenance of ecological interactions and for the provision of rare ecosystem services for the anthropic landscape, which is of great value to the city of Sao Paulo.