PieFedNephele WebDAV Server now supports deduplicated file storage.
lemmy.world/pictrs/image/1cf9bb97-f9b1-4be6-940…
submitted by hperrin@lemmy.world edited
https://hub.docker.com/r/sciactive/nephele
In the latest version of Nephele, you can now create a WebDAV server that deduplicates files that you add to it.
I created this feature because every night at midnight, my Minecraft world that my friends and I play on gets backed up. Our world has grown to about 5 GB, but every night, the same files get backed up over and over. It's a waste of space to store the same files again and again, but I want the ability to roll back our world to any day in the past.
So with this new feature of Nephele, I can upload the Minecraft backup and only the files that have changed will take up additional space. It's like having infinite incremental backups that never need a full backup after the first time, and can be accessed instantly.
Nephele will only delete a file from the file storage once all copies that share the same file contents have been deleted, so unlike with most incremental backup solutions, you can delete previous backups easily and regain space.
Edit: So, I think my post is causing some confusion. I should make it clear that my use case is specific for me. This is a general purpose deduplicating file server. It will take any files you give it and deduplicate them in its storage. It's not a backup system, and it's not a versioning system. My use case is only one of many you can use a deduplicating file server for.
That's a great feature! Currently I am way too invested in Nextcloud ecosystem to try anything else. But if I ever want to leave Nextcloud for a simple webdav server, this will be my consideration!
I looked into a bit more, and the fact that I can put it on top of an S3 compatible filesystem, with encryption and deduplication, is huge. I am seriously considering it as a backup system now.
Thank you. You just saved me from troubleshooting headaches with the other webdav. P.S., I uses the docker version with SSL cert on Unraid.
Cool, but you basically reinvented btrfs snapshots.
No, please research what deduplication is before commenting.
https://en.wikipedia.org/wiki/Data_deduplication
You might be thinking of incremental backups which also saves space but is not the same thing.
If you for example ran deduplication on a file server and a bunch of users uploaded the same files in multiple different directories, deduplication would remove all duplicate copies and just link them together. This has nothing to do with snapshots. btrfs might support deduplication but now this software does too. Your comment was completely unnecessary since not everything in the world can or should run btrfs
I am aware of the difference, but if you read the OP they are using it mainly for something that could also be done with btrfs.
Sure but maybe they don't want to use btrfs. Ever thought about that?
Their response further below clearly shows that they didn't bother researching btrfs or a similar filesystem that can do this.
Now of course they can do with their time what they want, but I am also free to point out that there are other ways (that are maybe more established) to reach the same goal.
Not at all. Btrfs snapshots:
They serve a very different purpose than a deduplicating file server. Now, there are other deduplicating file servers, but I don't know of any that are open source and run on Linux.
Uhm, I think you need to do better research as most of the above isn't true.
Can you tell me which is wrong?
Start with this to learn how snapshots work
https://fedoramagazine.org/working-with-btrfs-snapshots/
Then here the learn how to make automatic snapshots with retention
https://ounapuu.ee/posts/2022/04/05/btrfs-snapshots/
I do something very similar with zfs snapshots and deduplication on. I have one ever 5 mins and save 1 hr worth then save 24 hourlys every day and 1 day for a month etc
For backup to remote locations you can send a snapshot offsite
Having a separate tool do the work of making a snapshot doesn’t mean what I said is wrong. Snapshots are not automatic, with regard to btrfs. You can have a tool automatically make a snapshot, but btrfs won’t do it for you.
My overall point is that a deduplicating file server has very little in common with btrfs snapshots. The original commenter looked at my use case for my own deduplicating file server and assumed that the server was the same thing as my use case.
I think if they took the time to look at the server and see what it is actually doing, they would see that it is very different from btrfs.
Points 1,2,6,7 are wrong, and the others are partially wrong and/or can be easily solved with other existing tools.
Can you explain to me then:
If you’re using other tools on top of btrfs to implement a deduplicating file server, then you can’t say I reinvented btrfs snapshots, can you?
I don’t know how much clearer I can make the distinction between a copy on write file system and a deduplicating file server. They are completely different things for completely different purposes. The only thing they have in common is that they will deduplicate data, but a COW FS only deduplicates data under certain conditions. My server will deduplicate every file across its entire file store.
I get that people on Lemmy love to shit on other people’s accomplishments. I’ve never posted anything on here without it being criticized, but saying I “reinvented btrfs snapshots” is quite possibly the worst, most inaccurate take anyone has ever had on any of my posts.
Actually you seem to have reinvented Syncthing's versioning feature... or this.
Still great work.
So, to be clear, this is not a versioning system. I'm just kind of using it for that with my Minecraft backups. This is a deduplicating file server. It takes the files you give it and deduplicates them. Then, later, you can pull them out again. I am using it for backups, but it is also not a backup system.
I think I made it seem in my post like what I’m using it for is what it should be used for, or the only thing it can be used for. My use case is just one of many that you can use a deduplicating file server for.
Is there a link?
Also, how does this compare to something like Borg?
Here you go:
https://hub.docker.com/r/sciactive/nephele
So Borg is a backup solution. It will perform backups for you (as well as doing deduplication). This is just a deduplicating file server. It will take whatever you put in it, backups, videos, documents, etc. and deduplicate them. In my case, that's what I needed, not a complete backup solution like Borg.
Ah that makes much more sense. I think I crossed my wires. You mentioned backing up the Minecraft worlds and so I thought "deduplicated backups... so borg."
I appreciate your explanation.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
5 acronyms in this thread; the most compressed thread commented on today has 12 acronyms.
[Thread #811 for this sub, first seen 16th Jun 2024, 18:15] [[FAQ](http://decronym.xyz/)] [[Full list](http://decronym.xyz/acronyms/selfhosted@lemmy_world)] [[Contact](https://hachyderm.io/@Two9A)] [Source code]