In linux, how to delete all files EXCEPT the pattern *.txt?

I know how to delete all txt file under current directory by rm *.txt.
Does anyone know how to delete all files in current directory EXCEPT txt file?