How do i remove ‘,’ but only inside ‘”‘ with sed?

cat sample.txt
XY,"vol1",asdf,NFS,500,108,"10.99.0.0/16 10.2.0.0/16",unKnown
ZS,"vol2",asdf,NFS,"1,024","221.184","10.0.0.0/8",unKnown

sed 'SOMEMAGIC' sample.txt
XY,"vol1",asdf,NFS,500,108,"10.99.0.0/16 10.2.0.0/16",unKnown
ZS,"vol2",asdf,NFS,"1024","221.184","10.0.0.0/8",unKnown

so that 1,024 would be 1024

but i cannot simply do an:

sed 's/([0-9]),([0-9])/12/g'

since i would then hurt the:

500,108
->
500108

so i am searching for a solution which only replaces the “,” to nothing but ONLY inside a ‘ ” ‘