[Udpcast] Reliable multifile transfer over slow links, downed hosts
chris.curtis at riptidesoftware.com
Fri Oct 17 02:37:42 CEST 2003
I've been looking for an application for some time now that doesn't seem
to exist. The more I search, the more links lead to udpcast. My
scenario is basically this: I need to reliably transfer large files
(large as in >2GB) to multiple hosts over a slow WAN.
I've read some of the messages in this list regarding FEC and retries,
but I don't think this is sufficient. I've also looked into rsync batch
mode without much success (I expect the directory layouts to be
different). And in the background I keep thinking about BitTorrent.
What I would ideally want is something like the following:
'N' clients, each running a daemon (or inetd...)
Server Process connects via TCP to each client
Server tells client "Listen on Multicast port <foo>"
"I am going to send you XXXX bytes"
"Save this in a file called <bar>"
Assuming the client 'ok's these requests, the Server then begins
multicasting data packets. I'm thinking the packets are of variable
size, and are of some form:
Or the size is negotiated up front and not included in the packet.
However, what I also want is the client to record successful and
unsuccessful packet receives, such as "I got packets 1-1003"; "I got
When the multicast transmission is complete, the server asks each client
"What packets are you missing?" and the client responds in kind. The
server then retransmits the missed packets, and repeats doing this until
all the clients are happy.
Each client will have to store this state on disk, of course. And it
would be nice if (sometime way out in the future) resumption from a
server (process) outage would be possible, but that's not needed initially.
My question, then, is there anything like this available? If not, is
there is any interest in this here? Since I'm not familiar with the
code it would take me quite some time to find my way around. It may not
even be possible with the current code (specifically the packetization
portion). Or is this just outside the scope of udpcast?
More information about the Udpcast