DOI

The dramatic rise in amount of data worldwide over the past years is a critical issue for storage and backup systems with no obvious and simple solution available. One of the main techniques to effectively store large amounts of data is deduplication, based on eliminating redundant data, of which there is a lot. The most expensive stage of deduplication process is chunking, taking up to 90% of time. Many algorithms have emerged in the recent years, with the goal to optimize space savings and throughput of the process, stating better effectiveness each time. Despite there being so many chunking algorithms, there are very few systems that can be used to compare them with each other.In this paper we present ChunkFS, a tool to compare deduplication techniques that allows to easily integrate different chunking and hashing algorithms, as well as storage types, and gather the necessary metrics to determine the most suitable one. We use it to compare some of the best performing algorithms and find out that SuperCDC outperforms every other one in speed, but not in speed savings, although with many parameters that can be tuned, it hints that with further research using the tool better results can be achieved. Besides Content Defined Chunking algorithms, other techniques can be compared using ChunkFS, such as Frequency Based Chunking, but that is out of scope of this paper.
Язык оригиналаанглийский
Название основной публикации2025 37th Conference of Open Innovations Association (FRUCT)
ИздательInstitute of Electrical and Electronics Engineers Inc.
Страницы221-227
Число страниц7
ISBN (печатное издание)9789526524634
DOI
СостояниеОпубликовано - 14 мая 2025
СобытиеThe 37th FRUCT conference: FRUCT37 - UiT The Arctic University of Norway, Kufstein, Австрия
Продолжительность: 14 мая 202516 мая 2025
https://www.fruct.org/conferences/37/registration/

Серия публикаций

НазваниеConference of Open Innovation Association, FRUCT

конференция

конференцияThe 37th FRUCT conference
Сокращенное названиеFRUCT37
Страна/TерриторияАвстрия
ГородKufstein
Период14/05/2516/05/25
Сайт в сети Internet

ID: 137266822