How to remove comma from a csv file with quotes?

My Input is:

Name, Country, City
Jason, US, Memphis, "1,000"
David, US, Little Rock, "8,765,453"
"Karam, Sage", US, Nazareth, "4,678"
"David, simon", US, Chicago, "1,234"

I want output as:

Name, Country, City
Jason, US, Memphis, "1000"
David, US, Little Rock, "8765453"
"Karam, Sage", US, Nazareth, "4678"
"David, simon", US, Chicago, "1234"

I want the comma to be removed from only 4th column and not first column.