I encountered an OOM issue (this happens every-time I execute it) on running
grep -Fxvf file1 file2
file1 size: ~200MB
file2 size: ~300MB
number of records in each file: ~300K
avg records length: ~1K (only ASCII characters)
diff between two files is ~18K records
Available free memory: ~16GB
I tried with several different grep versions and in VM, WSL and also on a physical server, but got the same result.
Note that I ran the same command with only having few lines from both files to identify it doesn’t run into an infinite loop due to having some special character in the files and it was successful.
Is this normal?
I’m trying to output records from file2 which do not exist in file1.
I already resolved my requirement by awk in the same environment and got the output in less than about 10 seconds, but I’m wondering why the grep results in OOM.
I used the same command almost always when I needed to do querying same requirements and even I had compared two very big files like two files with ~2GB of size and each one ~90M records and records containing maximum ~20 ASCII characters without any issue in the same boxes.
I used GNU grep 2.7, 2.16 on SLES12, and GNU grep 3.7 on Ubnutu 22.04 in WSL.