Title is TLDR. More info about what I’m trying to do below.

My daily driver computer is Laptop with an SSD. No possibility to expand.

So for storage of lots n lots of files, I have an old, low resource Desktop with a bunch of HDDs plugged in (mostly via USB).

I can access Desktop files via SSH/SFTP on the LAN. But it can be quite slow.

And sometimes (not too often; this isn’t a main requirement) I take Laptop to use elsewhere. I do not plan to make Desktop available outside the network so I need to have a copy of required files on Laptop.

Therefor, sometimes I like to move the remote files from Desktop to Laptop to work on them. To make a sort of local cache. This could be individual files or directory trees.

But then I have a mess of duplication. Sometimes I forget to put the files back.

Seems like Laptop could be a lot more clever than I am and help with this. Like could it always fetch a remote file which is being edited and save it locally?

Is there any way to have Laptop fetch files, information about file trees, etc, located on Desktop when needed and smartly put them back after editing?

Or even keep some stuff around. Like lists of files, attributes, thumbnails etc. Even browsing the directory tree on Desktop can be slow sometimes.

I am not sure what this would be called.

Ideas and tools I am already comfortable with:

  • rsync is the most obvious foundation to work from but I am not sure exactly what would be the best configuration and how to manage it.

  • luckybackup is my favorite rsync GUI front end; it lets you save profiles, jobs etc which is sweet

  • freeFileSync is another GUI front end I’ve used but I am preferring lucky/rsync these days

  • I don’t think git is a viable solution here because there are already git directories included, there are many non-text files, and some of the directory trees are so large that they would cause git to choke looking at all the files.

  • syncthing might work. I’ve been having issues with it lately but I may have gotten these ironed out.

Something a little more transparent than the above would be cool but I am not sure if that exists?

Any help appreciated even just idea on what to web search for because I am stumped even on that.

  • Corgana@startrek.website
    2·
    6 months ago

    I have a very similar setup to you, and I use SyncThing without issue for the important files (which I keep in my Documents directory to make it easy to remember).

  • bloodfart@lemmy.ml
    1·
    6 months ago

    Hey I’m replying again directly to your post in the hopes that I can push against some of the advice you’re getting. My intent is to do an end run around arguing with the people making these suggestions because they’re very smart and made them for good reasons but their ideas aren’t necessarily good for you and I don’t want you to have to go through a troublesome recovery like I did and many people on the internet have.

    Do not under any circumstances set up raid or zpools for your data drives once you get them inside a case and on the pcie bus somehow.

    In these configurations accessing a file requires spinning up all the drives in the array or pool. Not only is that putting wear and tear on your drives, it increases the temperature of the case and draws much more power. Those conditions lead to drive failure. When your drive fails and you have a spare to use in its place, resilvering (the process of using extra data called parity to rebuild the contents of the failed drive on the spare one) will put those exact conditions on your remaining drives.

    For people like us, who may not have a hot spare, or great cooling, or an offsite backup, an array like that will set us up for failure rather than resilience.

    Please consider using mergerfs or something like it and a snapshot parity system like snapraid instead.

    There are very good use cases for the raid and zpool systems that have been brought up, but you aren’t there. I got there at moderate expense and moved away from them.

  • wargreymon2023@sopuli.xyz
    0·
    6 months ago
    1. wifi is less responsive than ethernet
    2. You don’t even need encryption on local connection(no need for ssh) if it is never global.
    3. if you have that many external HDD, I would rather get a NAS and call it a day, it has everything configured for you as well.
    4. I don’t know what big data you access on, get a larger SSD to store whatever you might visit often from your HDDs.
    • nickwitha_k (he/him)@lemmy.sdf.org
      1·
      6 months ago

      Strong disagreement on №2. That kinda thinking is how you get devices on your home network to join a malicious botnet without your knowledge and more identity theft. ALL network communications should assume that a malicious actor may be present and use encryption-in-transit for anything remotely approaching private, identifiable, or sensitive.

  • BearOfaTime@lemm.ee
    0·
    6 months ago

    1. What’s your network performance look like? 100mbit? Gigabit?

    2. External drives are terribly slow, USB doesn’t have great throughout. Also they’re unreliable, do you have backups? I’d look into making those drives internal (SATA) which has much better throughput. I use one external drive on USB3 for duplication, and it’s noticeably slower on file transfers, like 40%.

    3. For remote access to your files, look into Tailscale. You run it on your laptop and server (or any compatible device in your network), and it provides a virtual mesh network that functions like a LAN between devices.

    4. Syncthing is great, but it just keeps files in, entire folders of them. So if space is tight on the laptop, it won’t really help, not easily anyway.

    5. Resilio Sync has Selective Sync, where it can index a folder and store that index on any device participating in that sync job. Then you can select which files to sync at any time.

    • linuxPIPEpower@discuss.tchncs.deOP
      1·
      6 months ago

      1. In another comment I ran iperf3 Laptop (wifi) —> Desktop (ethernet) which was about 80-90MBits/s. Whereas Desktop —> OtherDesktop was in the 900-950 MBits/s range. So I think I can say the networking is fine enough when it’s all ethernet. Is there some other kind of benchmarking to do?

      2. Just posted a more detailed description of the desktops in this comment (4th paragraph). It’s not ideal but for now its what I have. I did actually take the time (gnome-disks benchmarking) to test different cables, ports, etc to find the best possible configuration. While there is an upper limit, if you are forced to use USB, this makes a big difference.

      3. Other people suggested ZeroTier or VPNs generally. I don’t really understand the role this component would be playing? I have a LAN and I really only want local access. Why the VPN?

      4. Ya, I have tried using syncthing for this before and it involves deleting stuff all the time then re-syncing it when you need it again. And you need to be careful not to accidentally delete while synced, which could destroy all files.

      5. Resilio I used it a long time ago. Didn’t realize it was still around! IIRC it was somewhat based on bittorrent with the idea of peers providing data to one another.