[Udpcast] udpcast scalability?

[Ramon Bastiaans]

George,

I'm afraid not really, well.. a little. Will explain why and what.

It seemed that the problem with us was the fact that udpcast uses 
synchronous writes to disk by default, and the fact that we pipe data 
through tar. 

For us, after some test scenarios (casting /dev/zero from sender to 
/dev/null on receiver, casting tarred data from sender to /dev/null on 
receiver), we found that the slow down/bottleneck was being caused by 
the synchronous writes. When setting udp-receiver to --nosync we had a 
(significant) speed increase.

We are running a Linux cluster of 275 machines which we install from a 
central server using SystemImager and udpcast.

Because SystemImager's image'ing tools dont compress or image the files, 
we need to cast an entire filesystem (lots of files) over the network. 
And because udpcast only supports sending/receiving a single file and 
writing to a single file descriptor, it can only write asynchronously to 
one file. Because of this we use tar to pipe all files through on both 
receiver and sender.

This is when the problem arose. When we didn't use tar, we could get 
high speeds and the (network/harddisk) hardware seemed to become the 
limiting factor. But only when using the --nosync writes. Because we use 
tar (which obviously has no --nosync option), now tar became the bottleneck.

Tar writes synchronous to disk by default and there seems to be no way 
to disable this. As an alternative Alain Knaff suggested using cpio, but 
we never came to this. This would require some massive changes to both 
server and client casting setup for SystemImager, and by then we got the 
speed up to 100 Mbps (still only a 10th of capability but better than 
before).

This was enough for us, since we are casting a image of approximately 2 
Gb. I can image it being a bigger problem when you're casting 20 Gb on a 
daily basis.

So, to cut a long story short:

Try adding the --nosync argument to your udp-receiver's, and see what 
happens.

When you don't pipe the data through an external program you should be 
able to get (very) fast speeds (at least we did). If you _do_ use tar or 
something to pipe the data through, we found that the maximum lies 
around 100 Mbps, because of previously explained reasons.

Another approach could be to first send the data as one big tar to your 
clients, and extract the tar after transmission. Or to try pipe'ing 
through cpio (haven't tried that one myself) in stead of tar.

Hope this helps you a bit. I know it can be a frustrating problem ;). 
Let me know how things work out for you.

Kind regards,
Ramon.

George Coulouris wrote:

> I've managed to get reliable transfers to all 85 machines at this 
> point (bad nic in the sender, I think).
>
> I'm still bottlenecking at 45Mbit or so, though, despite the fact that 
> all machines are on switched copper gigabit.
>
> If fec is turned on (either 8x8/128 or 16x4/128), receivers disconnect 
> throughout the transfer until only ~10 remain at the end.
>
> If fec is turned off, I get high retransmission rates (15-25%), but 
> the transfer at least succeeds to all hosts. Again, I'm sending about 
> 20G of data, which should take about an hour at 45Mbit. In practice, 
> it takes about an hour and a half, and sometimes close to two hours.
>
> For simple point-to-point tests with udpcast (between 1 and 5 
> receivers), I can achieve higher bandwidths reliably (sending 128 
> megabytes at 80mbit), both with and without fec.
>
> It sounds like my issue is similar to Ramon Bastiaan's:
> http://lll.lgl.lu/pipermail/udpcast/2004-September/000298.html
>
> Ramon, have you had any luck?