Dedupe in the real world
Posted by Sam
For those that haven't heard dedupe has made it's way into OpenSolaris 128a. You can download it at Genunix. To test what kind of savings we might expect to see in the real world I fired a VirtualBox with OpenSolaris 128a and turned on dedupe and compression (gzip level 6). Then I rsync'd 10 gigs worth of files from our staging server. The results? Dedupe dropped that 10 gigs to 6 gigs and compression dropped another 2 gigs which left us at 4 gigs. So dedupe gave us a 40% reduction and compression gave us another 20%. Not too shabby, especially when it takes about 10 seconds to enable it. A little bird told me that they are expecting dedupe in the Sun 7xxx series of storage by the end of the year. The Sun boxes are already the winner for best value and with that they jump even further out front!
Tags: opensolaris solaris deduplication zfs
Comments
about 1 month later Richard Lyle said
Hi Sam, Thanks for sending your blog link also. It's interesting that you DeDup'ed and then compressed data. Or were they both turned on at once? I'm expecting a technical whitepaper on the OpenStorage DeDup functionality and I will forward what we receive.
about 1 month later Sam Freiberg said
Thanks Richard. Looking forward to the whitepaper. Both dedupe and compression were turned on from the beginning.