Francis Scott Key(d. 1843) wrote the poem which became the lyrics to The Star Spangled Banner, he was dead long before the civil war started. His statues are being targeted now, because he owned slaves.
Ulysses S. Grant was the commander of the Army of the Potomac during the final years of the Civil War and was the one to accept Lee's Surrender ending the Civil War, he went on to serve two terms of PotUS where his administration was arguably one of the most ineffectual and corrupt in our history... But even his Statues are now under assault because he owned a slave for a little over a year before freeing him, and because his in-laws owned slaves.
Your point being? That those specific statues shouldn't have been toppled? Or are you using the toppling of those specific statues (which I'm undecided about) to somehow discredit the toppling of Confederate statues too?
On the balance, it seems to me that the vast majority of statues that get toppled thoroughly deserve it, and the examples you gave are a minority -- and I'm unclear whether those specific examples deserve it or not btw.
(But that's just my impression, is there anywhere a complete list of statues that have been toppled, so that we may judge the balance?)
Either way, as I've indicated already -- your government has lost legitimacy on this issue. Even if the government represented the majority of the people, that'd mean the majority has lost legitimacy on this issue: It no longer gets to decide which statues will remain and which won't. If a significant minority of the people disagree, the statue goes down: and for some it seems, "held slaves" is a big enough offense that it can't be excused by any other positive accomplishments.
In your case, those two were pointed out in case you didn't know they had no connection to the Confederacy and supporting its rebellion.
But with "they owned slaves" being grounds for removal. Then:
The National Anthem needs to be changed, it was written by a Slave Owner.
Washington DC needs to be renamed, it was named in honor of a Slave Owner.
The Washington monument needs to be renamed, it was named in honor of a Slave Owner.
The Declaration of Independence needs to meet a shredder, it was written by a Slave Owner and signed by many more slave owners.
The United State Constitution needs to meet a shredder, it was was signed by Slave Owners, and had significant input from them, in particular one James Madison.
The 1st Amendment and the rest of the Bill of Rights need to go as well, since Madison was involved in those too. So no more freedom of the press.
The Jefferson Memorial in Washington DC needs to be replaced, he owned slaves, can't be memorializing a slave owner.
The White House and the Capital Hill probably should be torn down, slave labor was likely involved in their initial construction and upkeep at times in the past.
The state of Washington needs to be renamed, as it was named to honor the first President of the United States, who was a slave owner.
Mount Rushmore needs to have two of its 4 heads removed(Washington and Jefferson), as they were slave owners.
Any statues of Benjamin Franklin need to be removed, anything that memorializes him needs to be removed, he owned slaves too.
Alexander Hamilton bought, sold, and traded slaves although it is unclear as whether or not he used any slaves himself. So to be safe, he cannot be memorialized either.
Jacksonville, Florida needs to be renamed, it was named in honor of Andrew Jackson, he owned slaves. Likewise, any statuary of him needs to be purged.
Also in that vein, this means:
The $1 dollar bill(George Washington) needs to be replaced,
If we still had $2 bills, they'd need replaced as Jefferson is on those, he owned slaves
the $10 Bill has Hamilton on it so it needs replaced because of slaves,
the $20 bill has Andrew Jackson on it and he owned slaves, needs replaced
the $50 bill had Grant on it, he owned a slave, needs replaced.
the $100 bill has Ben Franklin on it, he owned some slaves, needs replaced.
Of the first 12 presidents of the United States only 2 of them, John Adams and his son John Quincy Adams never owned a slave. Two more of the first 12 (Martin Van Buren and William Henry Harrison) didn't own slaves while serving as PotUS but did own slaves at other points in their life.
Fillmore(#13) and Pierce(#14) did not own slaves at any point, there appears to be some dispute on if Buchanan(#15) owned slaves or not, Lincoln(#16) did not own slaves at any time, but Johnson(#17) had owned slaves before, as did Grant(#18), Grant was the last president to have ever owned a slave at any time in their life.
This new purity test they wish to impose upon the country is a farce, it needs to stop.