[Udpcast] Reliable multifile transfer over slow links, downed hosts

Christopher Curtis chris.curtis at riptidesoftware.com
Fri Oct 17 02:37:42 CEST 2003


Hello,

I've been looking for an application for some time now that doesn't seem 
to exist.  The more I search, the more links lead to udpcast.  My 
scenario is basically this:  I need to reliably transfer large files 
(large as in >2GB) to multiple hosts over a slow WAN.

I've read some of the messages in this list regarding FEC and retries, 
but I don't think this is sufficient.  I've also looked into rsync batch 
mode without much success (I expect the directory layouts to be 
different).  And in the background I keep thinking about BitTorrent.

What I would ideally want is something like the following:
   'N' clients, each running a daemon (or inetd...)
     Server Process connects via TCP to each client
     Server tells client "Listen on Multicast port <foo>"
         "I am going to send you XXXX bytes"
         "Save this in a file called <bar>"

Assuming the client 'ok's these requests, the Server then begins 
multicasting data packets.  I'm thinking the packets are of variable 
size, and are of some form:

   [seq#|size|<data...>]

Or the size is negotiated up front and not included in the packet. 
However, what I also want is the client to record successful and 
unsuccessful packet receives, such as "I got packets 1-1003"; "I got 
packets 1005-9999".

When the multicast transmission is complete, the server asks each client 
"What packets are you missing?" and the client responds in kind.  The 
server then retransmits the missed packets, and repeats doing this until 
all the clients are happy.

Each client will have to store this state on disk, of course.  And it 
would be nice if (sometime way out in the future) resumption from a 
server (process) outage would be possible, but that's not needed initially.

My question, then, is there anything like this available?  If not, is 
there is any interest in this here?  Since I'm not familiar with the 
code it would take me quite some time to find my way around.  It may not 
even be possible with the current code (specifically the packetization 
portion).  Or is this just outside the scope of udpcast?

Thanks,
Chris



More information about the Udpcast mailing list