[Udpcast] udpcast scalability?
bastiaans at sara.nl
Thu Feb 24 10:06:23 CET 2005
I'm afraid not really, well.. a little. Will explain why and what.
It seemed that the problem with us was the fact that udpcast uses
synchronous writes to disk by default, and the fact that we pipe data
For us, after some test scenarios (casting /dev/zero from sender to
/dev/null on receiver, casting tarred data from sender to /dev/null on
receiver), we found that the slow down/bottleneck was being caused by
the synchronous writes. When setting udp-receiver to --nosync we had a
(significant) speed increase.
We are running a Linux cluster of 275 machines which we install from a
central server using SystemImager and udpcast.
Because SystemImager's image'ing tools dont compress or image the files,
we need to cast an entire filesystem (lots of files) over the network.
And because udpcast only supports sending/receiving a single file and
writing to a single file descriptor, it can only write asynchronously to
one file. Because of this we use tar to pipe all files through on both
receiver and sender.
This is when the problem arose. When we didn't use tar, we could get
high speeds and the (network/harddisk) hardware seemed to become the
limiting factor. But only when using the --nosync writes. Because we use
tar (which obviously has no --nosync option), now tar became the bottleneck.
Tar writes synchronous to disk by default and there seems to be no way
to disable this. As an alternative Alain Knaff suggested using cpio, but
we never came to this. This would require some massive changes to both
server and client casting setup for SystemImager, and by then we got the
speed up to 100 Mbps (still only a 10th of capability but better than
This was enough for us, since we are casting a image of approximately 2
Gb. I can image it being a bigger problem when you're casting 20 Gb on a
So, to cut a long story short:
Try adding the --nosync argument to your udp-receiver's, and see what
When you don't pipe the data through an external program you should be
able to get (very) fast speeds (at least we did). If you _do_ use tar or
something to pipe the data through, we found that the maximum lies
around 100 Mbps, because of previously explained reasons.
Another approach could be to first send the data as one big tar to your
clients, and extract the tar after transmission. Or to try pipe'ing
through cpio (haven't tried that one myself) in stead of tar.
Hope this helps you a bit. I know it can be a frustrating problem ;).
Let me know how things work out for you.
George Coulouris wrote:
> I've managed to get reliable transfers to all 85 machines at this
> point (bad nic in the sender, I think).
> I'm still bottlenecking at 45Mbit or so, though, despite the fact that
> all machines are on switched copper gigabit.
> If fec is turned on (either 8x8/128 or 16x4/128), receivers disconnect
> throughout the transfer until only ~10 remain at the end.
> If fec is turned off, I get high retransmission rates (15-25%), but
> the transfer at least succeeds to all hosts. Again, I'm sending about
> 20G of data, which should take about an hour at 45Mbit. In practice,
> it takes about an hour and a half, and sometimes close to two hours.
> For simple point-to-point tests with udpcast (between 1 and 5
> receivers), I can achieve higher bandwidths reliably (sending 128
> megabytes at 80mbit), both with and without fec.
> It sounds like my issue is similar to Ramon Bastiaan's:
> Ramon, have you had any luck?
More information about the Udpcast