[Udpcast] More on the corrupt file problem
kyle at kylecordes.com
Tue Nov 4 20:08:45 CET 2008
Alain Knaff wrote:
> The strange thing is, we do use udpcast for duplicating entire disks,
> most of which are larger than 50GB by now, and we never did notice any
> ill effect. A large piece of data missing in the middle would have been
> pretty obvious, but we've never have seen any of this so far.
By any chance do you typically use it like so on your large files?
udp-receiver | some-process ?
Contrary to my earlier findings, in ongoing testing I have found that if
I used it like this:
udp-receiver --file foo
I sometimes get bad results; and things like this:
udp-receiver --pipe "lzop -d" --file foo
also sometimes get bad results.
but I noticed that my real scripts do this:
udp-receiver | lzop -d | pg_restore
and I tested like this:
udp-receiver | lzop -d >foo
... and I get correct results. To 5 or 6 receivers. Every night.
Also, I found that having lzop (or other common compression tool) in the
loop acts as a guard against data integrity problems - if udp-receiver
skip or damages data, it would fail lzop's checksums and make the whole
Thus, it looks like there is some issue that comes in to play with
--file, but not when simply letting the data fall out on stdout.
I'm sitting the issue down for the moment, but later I may beat on it a
little more to try to track down the specifics of the failure.
I also think that perhaps the "--pipe" and "--file" features are
unnecessary; that udp-receiver would be better by being simpler, and
simply assume that the user will redirect the output where they need it.
More information about the Udpcast