Crews used a large crane to remove the statue from its pedestal as a small crowd gathered to watch.
Several work lorries were seen in the area but it was unclear where the statue could be taken.
The White House plan came after a protest over the statue led to police swinging batons and demonstrators hurling frozen water bottles, fireworks and other projectiles at officers.
It also stemmed from a shooting during a funeral that wounded 15 people.
A collection of activist groups had filed a lawsuit on Wednesday, seeking to block federal agents from interfering in or policing protests.
State officials in Oregon had sued for similar requests following the arrival of federal law enforcement due to nearly two months of protests in Portland since the death of George Floyd.
