How to stop usbfs claiming my device?

I’m trying to passthrough my usb device to guest os running on qemu. So I’ve added these arguments to my qemu call:

-device usb-host,vendorid=0x05ac,productid=0x12a8,guest-reset=false

But it won’t actually work and it cant’t, because usbfs keeps blocking my device (from dmesg):

usbfs: interface 1 claimed by usbfs while ‘qemu-system-x86’ sets
config

Qemu keeps trying to claim device and it keeps reporting device is busy. How do I force usbfs to release my device? There’s no any kernel module named usbfs:

rmmod: ERROR: Module usbfs is not currently loaded

There’s also no any process called usbfs. Any ideas? My OS is Debian/LMDE