Mismanaged plastic waste is transported via rivers or city drains into the ocean where it accumulates in coastal sediments, ocean gyres and the deep ocean. Plastic harms marine biota and may ultimately return to humans via the food chain. Private initiatives proposing to collect plastic from the sea and rivers have gained widespread attention, especially in the media. However, few of these methods are proven concepts and it remains unclear how effective they are. Here we estimate the amount of plastic in the global surface ocean to assess the long-term legacy of plastic mass production, calculate the time required to clean up the oceans with river barriers and clean up devices, and explore the fate of collected plastic waste. We find that the projected impact of both single and multiple clean up devices is very modest. A significant reduction of plastic debris in the ocean can be only achieved with collection at rivers or with a combination of river barriers and clean up devices. We also show that the incineration and production of plastic has a significant long-term effect on the global atmospheric carbon budget. We conclude that a combination of reduced plastic emissions and reinforced collection is the only way to rid the ocean of plastic waste.