I had the same problem when running dedup on FreeNAS under ZFS.

In the end, I decided that the RAM hit and risk was not worth the savings.