At least 50 indigenous groups spread across lowland South America remain isolated and have only intermittent and mostly hostile interactions with the outside world. Except in emergency situations, the current policy of governments in Brazil, Colombia, and Peru towards isolated tribes is a “leave them alone” strategy, in which isolated groups are left uncontacted. However, these no-contact policies are based on the assumption that isolated populations are healthy and capable of persisting in the face of mounting external threats, and that they can maintain population viability in the long-term. Here, we test this assumption by tracking the sizes and movements of cleared horticultural areas made by 8 isolated groups over the last 10–14 years. We used deforestation data derived from remote sensing Landsat satellite sensors to identify clearings, and those were then validated and assessed with high-resolution imagery. We found only a single example of a relatively large and growing population (c. 50 cleared ha and 400 people), whereas all of the other 7 groups exhibited much smaller villages and gardens with no sizable growth through time. These results indicated that the smaller groups are critically endangered, and it prompts an urgent re-thinking of policies toward isolated populations, including plans for well-organized contacts that may help save lives and rescue isolated indigenous populations from imminent extinction.