From cjwjavier at gmail.com Fri Oct 17 05:50:28 2008 From: cjwjavier at gmail.com (javier.chen) Date: Fri, 17 Oct 2008 11:50:28 +0800 Subject: [Udpcast] there are some errors in udpcdialog, can u help me? Message-ID: In file included from udpcdialog//cursWrap.c:18: udpcdialog//dialog.h:32:20: error: curses.h: No such file or directory In file included from udpcdialog//cursWrap.c:18: udpcdialog//dialog.h:124: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'use_colors' udpcdialog//dialog.h:125: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'use_shadow' udpcdialog//dialog.h:128: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'attributes' udpcdialog//dialog.h:141: error: expected ')' before '*' token udpcdialog//dialog.h:145: error: expected ')' before '*' token udpcdialog//dialog.h:146: error: expected ')' before '*' token udpcdialog//dialog.h:147: error: expected ')' before '*' token udpcdialog//dialog.h:150: error: expected ')' before '*' token udpcdialog//cursWrap.c: In function 'udpc_clearScreen': udpcdialog//cursWrap.c:21: warning: implicit declaration of function 'attr_clear' udpcdialog//cursWrap.c:21: error: 'stdscr' undeclared (first use in this function) udpcdialog//cursWrap.c:21: error: (Each undeclared identifier is reported only once udpcdialog//cursWrap.c:21: error: for each function it appears in.) udpcdialog//cursWrap.c:21: error: 'LINES' undeclared (first use in this function) udpcdialog//cursWrap.c:21: error: 'COLS' undeclared (first use in this function) udpcdialog//cursWrap.c:21: error: 'attributes' undeclared (first use in this function) udpcdialog//cursWrap.c:22: warning: implicit declaration of function 'move' udpcdialog//cursWrap.c:23: warning: implicit declaration of function 'refresh' make[1]: *** [udpcdialog//cursWrap.o] Error 1 make: *** [udpcdialog/] Error 2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alain at knaff.lu Fri Oct 17 06:47:40 2008 From: alain at knaff.lu (Alain Knaff) Date: Fri, 17 Oct 2008 06:47:40 +0200 Subject: [Udpcast] there are some errors in udpcdialog, can u help me? In-Reply-To: References: Message-ID: <48F818EC.9010305@knaff.lu> javier.chen wrote: > In file included from udpcdialog//cursWrap.c:18: > udpcdialog//dialog.h:32:20: error: curses.h: No such file or directory ^^^^^^^^ You need to have the headers for curses or ncurses installed. (In Kubuntu, this is in package libncurses5-dev) Regards, Alain From cjwjavier at gmail.com Fri Oct 17 11:33:51 2008 From: cjwjavier at gmail.com (javier.chen) Date: Fri, 17 Oct 2008 17:33:51 +0800 Subject: [Udpcast] hi all, what can i do now? Message-ID: I have compiled a good busybox(the menu and udpcast added), and also compiled a .rom file for my etherboot. And i want to make udp-sender/receiver run before linux start. Then what should I do? To put busybox into rom? or other things? Please. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alain at knaff.lu Fri Oct 17 12:01:21 2008 From: alain at knaff.lu (Alain Knaff) Date: Fri, 17 Oct 2008 12:01:21 +0200 Subject: [Udpcast] hi all, what can i do now? In-Reply-To: References: Message-ID: <48F86271.3050503@knaff.lu> javier.chen wrote: > I have compiled a good busybox(the menu and udpcast added), and also > compiled a .rom file for my etherboot. > And i want to make udp-sender/receiver run before linux start. > Then what should I do? To put busybox into rom? or other things? > Please. A .rom file is used to enable your network card to boot from the network. You need to flash it into the card's EPROM, you should either check your network card's documentation, or the etherboot website for instructions on how to do this. Please note however, that nowadays, most network cards are already to boot from network out of the box, so you might not need this step, and the .rom file at all. Once you have a card capable of booting from the network, you need to set up your server. 1. Make sure you've got a dhcp server, and that it's configured correctly 2. Make sure that you've got a tftp server . Prefer to use tftpd-hpa rather than plain tftpd 3. Set a filename to boot from in your dhcp.conf by adding the following instructions, and restart your dhcp server: next-server 1.2.3.4; filename "pxelinux.0"; where 1.2.3.4 is the IP address of your tftp server. If the tftp server and the dhcp server reside on the same machine, you can drop that line. 4. Copy pxelinux.0 (from syslinux) to the /var/lib/tftpboot directory on your tftp server. 5. Copy linux and initrd (from http://udpcast.linux.lu/current) to the same directory 6. make a /var/lib/tftpboot/pxelinux.cfg directory and put a default file in it with the following contents (between, but not including, the lines): ------- DEFAULT udpcast LABEL udpcast kernel linux append initrd=initrd PROMPT 0 TIMEOUT 0 -------- Regards, Alain From cjwjavier at gmail.com Mon Oct 20 05:19:52 2008 From: cjwjavier at gmail.com (javier.chen) Date: Mon, 20 Oct 2008 11:19:52 +0800 Subject: [Udpcast] how to... Message-ID: Hi, alain, sorry, i do not have a clear expression of what I mean. My mean: power on(my client computer) -> show a menu -> choose udp-receiver, then i can receive the large file from server. What can i do? Using udpcast instead of PXE? And how to? Regard, Javier.Chen -------------- next part -------------- An HTML attachment was scrubbed... URL: From alain at knaff.lu Mon Oct 20 06:20:10 2008 From: alain at knaff.lu (Alain Knaff) Date: Mon, 20 Oct 2008 06:20:10 +0200 Subject: [Udpcast] how to... In-Reply-To: References: Message-ID: <48FC06FA.5070501@knaff.lu> javier.chen wrote: > Hi, alain, sorry, i do not have a clear expression of what I mean. > > My mean: power on(my client computer) -> show a menu -> > choose udp-receiver, then i can receive the large file from server. Yes, that's supposed how it is supposed to work... > > What can i do? > Using udpcast instead of PXE? PXE is one of several ways to boot udpcast. Other ways are CDROM or floppy. So it's not "instead", but rather "with". > And how to? What is your question exactly, at which step are you stuck? Or maybe try in Spanish (off-list...) if you are more comfortable that way. Two of my colleagues at work speak Spanish, maybe they can translate. > Regard, > Javier.Chen Regards, Alain From stephane.boireau at free.fr Mon Oct 20 11:58:22 2008 From: stephane.boireau at free.fr (Stephane Boireau) Date: Mon, 20 Oct 2008 11:58:22 +0200 Subject: [Udpcast] Udpcast 2008-09-14: fullbox??? Message-ID: <200810201158.22409.stephane.boireau@free.fr> Hi, It seems, there is a trouble with building new image using 20080914 version. # ./makeImage -k kernel/vmlinuz-2.6.26.3udpcast --fullbox -c /home/steph/udpcast/udpcast_20081020.iso 24306 blocs Kversion=2.6.26.3udpcast Using ramdisk 01:00 I: -input-charset not specified, using iso-8859-15 (detected in locale settings) Size of boot image is 4 sectors -> No emulation Total translation table size: 2048 Total rockridge attributes bytes: 576 Total directory bytes: 0 Path table size(bytes): 10 Max brk space used 0 3055 extents written (5 MB) # Booting this ISO, no fullbox, not even ls or vi: # ls ls: not found # My host system is Debian etch (but I use alien to convert RPM to DEB because deb versions in http://udpcast.linux.lu/source.html require udpcast dépend de libc6 (>= 2.4-1) ; cependant : La version de libc6 sur le système est 2.3.6.ds1-13etch7. ) Regards. -- Stephane Boireau. From stephane.boireau at free.fr Thu Oct 23 11:29:51 2008 From: stephane.boireau at free.fr (Stephane Boireau) Date: Thu, 23 Oct 2008 11:29:51 +0200 Subject: [Udpcast] Udpcast 2008-09-14: fullbox??? In-Reply-To: <200810201158.22409.stephane.boireau@free.fr> References: <200810201158.22409.stephane.boireau@free.fr> Message-ID: <200810231129.51669.stephane.boireau@free.fr> Hi, Le lundi 20 octobre 2008 11:58, vous avez écrit : > It seems, there is a trouble with building new image using 20080914 > version. I apologize. It seems ok using ubuntu and udpcast 20080914 debs. I'll try to understand what goes wrong with debian etch and rpm->alien->deb... Regards. -- Stephane. > # ./makeImage -k > kernel/vmlinuz-2.6.26.3udpcast --fullbox -c > /home/steph/udpcast/udpcast_20081020.iso 24306 blocs > Kversion=2.6.26.3udpcast > Using ramdisk 01:00 > I: -input-charset not specified, using iso-8859-15 (detected in locale > settings) > Size of boot image is 4 sectors -> No emulation > Total translation table size: 2048 > Total rockridge attributes bytes: 576 > Total directory bytes: 0 > Path table size(bytes): 10 > Max brk space used 0 > 3055 extents written (5 MB) > # > > Booting this ISO, no fullbox, not even ls or vi: > > # ls > ls: not found > # > > My host system is Debian etch (but I use alien to convert RPM to DEB > because deb versions in http://udpcast.linux.lu/source.html require > udpcast dépend de libc6 (>= 2.4-1) ; cependant : > La version de libc6 sur le système est 2.3.6.ds1-13etch7. > ) > > > Regards. -- Stephane Boireau. From mikes at kuentos.guam.net Tue Oct 28 16:59:36 2008 From: mikes at kuentos.guam.net (Michael D. Setzer II) Date: Wed, 29 Oct 2008 01:59:36 +1000 Subject: [Udpcast] Quesiton on Kernel builds? Message-ID: <490736E8.7834.63A66A6@mikes.kuentos.guam.net> I just build some new udpcast cd images, and noted that the ATA drives are now showing up as sda instead of (hda) in the older version.The same issue occurred some time ago with Fedora. Don't recall the exact version where it changed. I maintain the g4l disk imaging project, and include the udp-sender and udp-reciever on it, but also recommend using udpcast for doing many machines at once. I actually create images on my servers via g4l and ftp, but the use udpcast to image the lab machines from one to many. For the g4l, I've build the kernel with everything built into the kernel, which makes a single file, and the cd has a number of various kernels since some work with one setup of hardware, while others may not. Except for one scis_wait something modual that always builds as a modual, it is just the single bzImage files that I rename. Have currently created images up the 2.6.27.4. This seems to work fine, and make the cd image easy to add, but it still has the ATA disk coming up as (hda). Not a problem, but I've had a couple of users say they are getting very slow disk performance on some setups. On my own machine, I seem to see no difference in speed, but I'm wondering if on some systems, it is selecting a slower compatible disk driver or something. At one point, I had a user with a SIS5513 controller that required a kernel patch to get higher speeds, but believe that was resolved in an updated to the kernel source that was suppose to get the correct bus speed. Wish I had more info on building kernels. Thanks. +----------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mikes at kuentos.guam.net mailto:msetzerii at gmail.com http://www.guam.net/home/mikes Guam - Where America's Day Begins +----------------------------------------------------------+ http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489) BOINC at HOME CREDITS SETI 6,681,217.6794 | EINSTEIN 2,067,250.6609 | ROSETTA 659,343.5601 From kyle at kylecordes.com Wed Oct 29 20:38:13 2008 From: kyle at kylecordes.com (Kyle Cordes) Date: Wed, 29 Oct 2008 14:38:13 -0500 Subject: [Udpcast] Confusing, troubling UDPcast results Message-ID: <4908BBA5.7090605@kylecordes.com> I'm using the latest (20080914) UDPcast, which I compiled from source on Ubuntu 8.04 on x64 (xeon). I get the same results I describe here, when I use the ancient (2004) UDPCast available in the Ubuntu packaging system. I also get the same results I describe here, whether I use "--file" or >file. I am transmitting data about 51GB in size from one machine to several others; it is a database backup that I pipe from pg_dump with something like this: pg_dump yadayada | udp-sender --no-kbd --autostart 90 and I receive it on 3 other machines with this: udp-receiver --portbase 10200 --file filenamehere On the sender, the output ends like so: bytes= 54 682 988 148 re-xmits=0166670 ( 0.4%) slice=0032 73 709 551 615 - 2 Transfer complete. Disconnecting #0 (ip-address-redacted) Disconnecting #1 (ip-address-redacted) Disconnecting #2 (ip-address-redacted) On receiver 1, it ends like so: bytes= 54 682 988 148 (123.12 Mbps) 54 682 848 884 Transfer complete. the file is 54682848884 bytes long. On receiver 2, it ends like so: bytes= 54 682 988 148 (123.11 Mbps) 54 682 848 884 Transfer complete. the file is 54682848884 bytes long. On receiver 3, it ends like so: bytes= 54 682 988 148 (123.13 Mbps) 54 682 938 996 Transfer complete. the file is 54682938996 bytes long. On this particular run, I got "lucky" in that two of the 3 receivers got the same file length; most of the time, 3 receivers end up with files of 3 different lengths. I think the correct length is 54 682 988 148, meaning that 0 of the 3 got the file delivered correctly. In the output, udp-receiver appears to know both the correct length and the actual written length, yet not be bothered by them being different (!!). I find this quite surprising and rather disturbing... I had planned to wrap an md5 mechanism around UDPcast somehow as an end-to-end check of the contents, but it's pointless to do that if it doesn't even get the correct *length* of the data. Any help much appreciated, as I find UDPcast to be a fantastic idea and nicely implemented tool, aside from this problem. Could there be some issue in which udp-receiver is confused between what go to the file/stdout (data payload) vs to stderr (absolutely everything else) ? -- Kyle Cordes http://kylecordes.com From kyle at kylecordes.com Wed Oct 29 22:48:01 2008 From: kyle at kylecordes.com (Kyle Cordes) Date: Wed, 29 Oct 2008 16:48:01 -0500 Subject: [Udpcast] Reducing frequency of status messages to stderr Message-ID: <4908DA11.9070304@kylecordes.com> Currently, udp-sender and -receiver send many updates per second to stderr describing the current progress and data rate. I'd like to reduce that to perhaps one update per second. Any suggestions on where in the source code to make such a change? -- Kyle Cordes http://kylecordes.com From kyle at kylecordes.com Wed Oct 29 23:22:59 2008 From: kyle at kylecordes.com (Kyle Cordes) Date: Wed, 29 Oct 2008 17:22:59 -0500 Subject: [Udpcast] checking write() return value Message-ID: <4908E243.9090908@kylecordes.com> Trying to track down the problem I posted about earlier, I made a few tiny changes, including an assert-style check on write(). This case this code worries about, does not occur in my test with a ~50 GB file. Still, code like this is generally a good idea, rather than assuming that write() wrote the expected amount of data. Kyle diff --git a/receiver-diskio.c b/receiver-diskio.c index 2076ba2..500ac2a 100644 --- a/receiver-diskio.c +++ b/receiver-diskio.c @@ -43,14 +43,18 @@ int writer(struct fifo *fifo, int outFile) { * liberate small chunks one by one rather than attempt to * write out a bigger chunk and block reception for too * long */ - if (bytes > 128 * 1024) + if (bytes > 64 * 1024) bytes = 64 * 1024; - bytes = write(outFile, fifo->dataBuffer + pos, bytes); - if(bytes < 0) { + int bytesWritten = write(outFile, fifo->dataBuffer + pos, bytes); + if(bytesWritten < 0) { perror("write"); exit(1); } + if(bytesWritten != bytes) { + perror("write, bytesWritten != bytes"); + exit(1); + } pc_consumed(fifo->data, bytes); pc_produce(fifo->freeMemQueue, bytes); } -- Kyle Cordes http://kylecordes.com From alain at knaff.lu Thu Oct 30 14:48:24 2008 From: alain at knaff.lu (Alain Knaff) Date: Thu, 30 Oct 2008 14:48:24 +0100 Subject: [Udpcast] Quesiton on Kernel builds? In-Reply-To: <490736E8.7834.63A66A6@mikes.kuentos.guam.net> References: <490736E8.7834.63A66A6@mikes.kuentos.guam.net> Message-ID: <4909BB28.1010606@knaff.lu> Michael D. Setzer II wrote: > I just build some new udpcast cd images, and noted that the ATA drives are now showing up > as sda instead of (hda) in the older version.The same issue occurred some time ago with > Fedora. Don't recall the exact version where it changed. Yes, most recent distributions do behave like this. Starting from kernel 2.6.24, I no longer managed to compile it to use the "old" names, so I had to move to sda. > I maintain the g4l disk imaging project, and include the udp-sender and udp-reciever on it, > but also recommend using udpcast for doing many machines at once. I actually create > images on my servers via g4l and ftp, but the use udpcast to image the lab machines from > one to many. > > For the g4l, I've build the kernel with everything built into the kernel, which makes a single > file, and the cd has a number of various kernels since some work with one setup of > hardware, while others may not. Except for one scis_wait something modual that always > builds as a modual, it is just the single bzImage files that I rename. Have currently created > images up the 2.6.27.4. This seems to work fine, and make the cd image easy to add, but it > still has the ATA disk coming up as (hda). Not a problem, but I've had a couple of users say > they are getting very slow disk performance on some setups. On my own machine, I seem to > see no difference in speed, but I'm wondering if on some systems, it is selecting a slower > compatible disk driver or something. At one point, I had a user with a SIS5513 controller that > required a kernel patch to get higher speeds, but believe that was resolved in an updated to > the kernel source that was suppose to get the correct bus speed. > > Wish I had more info on building kernels. > > Thanks. Check http://udpcast.linux.lu/mkimagedoc.html#kernelCompile for instructions Alain From kyle at kylecordes.com Thu Oct 30 15:15:35 2008 From: kyle at kylecordes.com (Kyle Cordes) Date: Thu, 30 Oct 2008 09:15:35 -0500 Subject: [Udpcast] Confusing, troubling UDPcast results In-Reply-To: <4908BBA5.7090605@kylecordes.com> References: <4908BBA5.7090605@kylecordes.com> Message-ID: <4909C187.5030208@kylecordes.com> Kyle Cordes wrote: > I'm using the latest (20080914) UDPcast, which I compiled from source on > Ubuntu 8.04 on x64 (xeon). I get the same results I describe here, when > I use the ancient (2004) UDPCast available in the Ubuntu packaging > system. I also get the same results I describe here, whether I use > "--file" or >file. An update, though I don't know if anyone is listening :-) I found that for my 50 GB+ files, udp-receiver also delivers the wrong number of bytes to its --pipe, thus on the sending side, something like this: udp-sender --file file1 On the receiving side: udp-receiver --pipe "tee file2" >/dev/null Results in file2 having a different length than file1. Both are around 52GB in this particular test. Perhaps there is a clue to the problem in there somewhere. -- Kyle Cordes http://kylecordes.com From mikes at kuentos.guam.net Thu Oct 30 16:08:26 2008 From: mikes at kuentos.guam.net (Michael D. Setzer II) Date: Fri, 31 Oct 2008 01:08:26 +1000 Subject: [Udpcast] Quesiton on Kernel builds? In-Reply-To: <4909BB28.1010606@knaff.lu> References: <490736E8.7834.63A66A6@mikes.kuentos.guam.net>, <4909BB28.1010606@knaff.lu> Message-ID: <4909CDEA.8032.814DD8B@mikes.kuentos.guam.net> On 30 Oct 2008 at 14:48, Alain Knaff wrote: Date sent: Thu, 30 Oct 2008 14:48:24 +0100 From: Alain Knaff To: "Michael D. Setzer II" Copies to: udpcast at udpcast.linux.lu Subject: Re: [Udpcast] Quesiton on Kernel builds? > Michael D. Setzer II wrote: > > I just build some new udpcast cd images, and noted that the ATA drives are now showing up > > as sda instead of (hda) in the older version.The same issue occurred some time ago with > > Fedora. Don't recall the exact version where it changed. > > Yes, most recent distributions do behave like this. Starting from kernel > 2.6.24, I no longer managed to compile it to use the "old" names, so I had > to move to sda. How? I have build kernels all the way to 2.6.27.4, but it still shows the ATA devices as (hda). > > > I maintain the g4l disk imaging project, and include the udp-sender and udp-reciever on it, > > but also recommend using udpcast for doing many machines at once. I actually create > > images on my servers via g4l and ftp, but the use udpcast to image the lab machines from > > one to many. > > > > For the g4l, I've build the kernel with everything built into the kernel, which makes a single > > file, and the cd has a number of various kernels since some work with one setup of > > hardware, while others may not. Except for one scis_wait something modual that always > > builds as a modual, it is just the single bzImage files that I rename. Have currently created > > images up the 2.6.27.4. This seems to work fine, and make the cd image easy to add, but it > > still has the ATA disk coming up as (hda). Not a problem, but I've had a couple of users say > > they are getting very slow disk performance on some setups. On my own machine, I seem to > > see no difference in speed, but I'm wondering if on some systems, it is selecting a slower > > compatible disk driver or something. At one point, I had a user with a SIS5513 controller that > > required a kernel patch to get higher speeds, but believe that was resolved in an updated to > > the kernel source that was suppose to get the correct bus speed. > > > > Wish I had more info on building kernels. > > > > Thanks. > > > Check http://udpcast.linux.lu/mkimagedoc.html#kernelCompile for instructions I had looked at that before, but the system creates modules, and would require, and with the current setup of having about 10 kernels would require making many directories with these files, and since it resides in the filesystem, whereas the kernels with everything build in only take about 5 - 6 M that are in the isolinux directory, and don't take space on the ramdisk file system.... Thanks for the info, will have to look into things a little more. > > Alain +----------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mikes at kuentos.guam.net mailto:msetzerii at gmail.com http://www.guam.net/home/mikes Guam - Where America's Day Begins +----------------------------------------------------------+ http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489) BOINC at HOME CREDITS SETI 6,693,814.7866 | EINSTEIN 2,075,050.6209 | ROSETTA 663,492.9235 From alain at knaff.lu Thu Oct 30 16:38:39 2008 From: alain at knaff.lu (Alain Knaff) Date: Thu, 30 Oct 2008 16:38:39 +0100 Subject: [Udpcast] Quesiton on Kernel builds? In-Reply-To: <4909CDEA.8032.814DD8B@mikes.kuentos.guam.net> References: <490736E8.7834.63A66A6@mikes.kuentos.guam.net>, <4909BB28.1010606@knaff.lu> <4909CDEA.8032.814DD8B@mikes.kuentos.guam.net> Message-ID: <4909D4FF.2010506@knaff.lu> Michael D. Setzer II wrote: > I had looked at that before, but the system creates modules, and would require, and with the > current setup of having about 10 kernels would require making many directories with these > files, and since it resides in the filesystem, whereas the kernels with everything build in only > take about 5 - 6 M that are in the isolinux directory, and don't take space on the ramdisk file > system.... Well, theoretically, you could take my config file (to compile a kernel with modules), boot it, find out which modules you need. For the modules you need, replace the =m with =y to compile them in For the others remove them (by saying # CONFIG_FEATURE_XYZ is not set) > Thanks for the info, will have to look into things a little more. > >> Alain Regards, Alain From mikes at kuentos.guam.net Thu Oct 30 16:49:48 2008 From: mikes at kuentos.guam.net (Michael D. Setzer II) Date: Fri, 31 Oct 2008 01:49:48 +1000 Subject: [Udpcast] Quesiton on Kernel builds? In-Reply-To: <4909D4FF.2010506@knaff.lu> References: <490736E8.7834.63A66A6@mikes.kuentos.guam.net>, <4909CDEA.8032.814DD8B@mikes.kuentos.guam.net>, <4909D4FF.2010506@knaff.lu> Message-ID: <4909D79C.16359.83ABF47@mikes.kuentos.guam.net> On 30 Oct 2008 at 16:38, Alain Knaff wrote: Date sent: Thu, 30 Oct 2008 16:38:39 +0100 From: Alain Knaff To: "Michael D. Setzer II" Copies to: udpcast at udpcast.linux.lu Subject: Re: [Udpcast] Quesiton on Kernel builds? > Michael D. Setzer II wrote: > > I had looked at that before, but the system creates modules, and would require, and with the > > current setup of having about 10 kernels would require making many directories with these > > files, and since it resides in the filesystem, whereas the kernels with everything build in only > > take about 5 - 6 M that are in the isolinux directory, and don't take space on the ramdisk file > > system.... > > Well, theoretically, you could take my config file (to compile a kernel > with modules), boot it, find out which modules you need. > Thanks. If it were just for my machines, that would be no problem, but I get about 400 or so downloads of the program from sourceforge and freshmeat sites each day, so it kind of needs everything? With my hardware, I get similar results with hdparm with the g4l (hda) or the fedora (sda). But had one user that was getting very different results. Didn't get more info, so not sure what was causing it. My thought was that it might be using an older compatible mode of the controller, but not sure how it determines which driver to try first.. Don't know if that many people are really using the program, or just downloading it, but it seems to work for most I guess, but if I can make it better I would like to. > For the modules you need, replace the =m with =y to compile them in > For the others remove them (by saying # CONFIG_FEATURE_XYZ is not set) > > > Thanks for the info, will have to look into things a little more. > > > >> Alain > > Regards, > > Alain +----------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mikes at kuentos.guam.net mailto:msetzerii at gmail.com http://www.guam.net/home/mikes Guam - Where America's Day Begins +----------------------------------------------------------+ http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489) BOINC at HOME CREDITS SETI 6,693,814.7866 | EINSTEIN 2,075,050.6209 | ROSETTA 663,492.9235 From alain at knaff.lu Thu Oct 30 18:59:36 2008 From: alain at knaff.lu (Alain Knaff) Date: Thu, 30 Oct 2008 18:59:36 +0100 Subject: [Udpcast] Quesiton on Kernel builds? In-Reply-To: <4909D79C.16359.83ABF47@mikes.kuentos.guam.net> References: <490736E8.7834.63A66A6@mikes.kuentos.guam.net>, <4909CDEA.8032.814DD8B@mikes.kuentos.guam.net>, <4909D4FF.2010506@knaff.lu> <4909D79C.16359.83ABF47@mikes.kuentos.guam.net> Message-ID: <4909F608.5000405@knaff.lu> Michael D. Setzer II wrote: > On 30 Oct 2008 at 16:38, Alain Knaff wrote: > > Date sent: Thu, 30 Oct 2008 16:38:39 +0100 > From: Alain Knaff > To: "Michael D. Setzer II" > Copies to: udpcast at udpcast.linux.lu > Subject: Re: [Udpcast] Quesiton on Kernel builds? > >> Michael D. Setzer II wrote: >>> I had looked at that before, but the system creates modules, and would require, and with the >>> current setup of having about 10 kernels would require making many directories with these >>> files, and since it resides in the filesystem, whereas the kernels with everything build in only >>> take about 5 - 6 M that are in the isolinux directory, and don't take space on the ramdisk file >>> system.... >> Well, theoretically, you could take my config file (to compile a kernel >> with modules), boot it, find out which modules you need. >> > > Thanks. If it were just for my machines, that would be no problem, but I get about 400 or so > downloads of the program from sourceforge and freshmeat sites each day, so it kind of > needs everything? With my hardware, I get similar results with hdparm with the g4l (hda) or > the fedora (sda). But had one user that was getting very different results. Didn't get more > info, so not sure what was causing it. My thought was that it might be using an older > compatible mode of the controller, but not sure how it determines which driver to try first.. > > Don't know if that many people are really using the program, or just downloading it, but it > seems to work for most I guess, but if I can make it better I would like to. Well, you could theoretically enable _all_ modules (by substituting =m for =y everywhere) in my kernel config file. You'll get a huge kernel (somewhat defeating the point of the customization), but it will work for all machines. There might however be some stuff that is useless for a udpcast-like program, such as sound drivers, etc. If you notice any of these, you can remove those Alain From kyle at kylecordes.com Thu Oct 30 20:30:35 2008 From: kyle at kylecordes.com (Kyle Cordes) Date: Thu, 30 Oct 2008 14:30:35 -0500 Subject: [Udpcast] More on the corrupt file problem Message-ID: <490A0B5B.2020903@kylecordes.com> To try to track down the udpcast corrupt file problem, I ran some more tests. This time I used a ~50GB file, a sender, and only 1 receiver. side bytes sender: 53687091200 receiver: 53686091776 In all of my large-file runs, udp-receiver comes up a bit "short", it missing some of the data, never "long". I created a 50 GB test file with predictable text data in it, suing this ugly little program: #include // 16 bytes per entry. // int main(void) { long long gb = 1024 * 1024 * 1024; long long m = 50 * gb; long long i; for(i = 0; i References: <490A0B5B.2020903@kylecordes.com> Message-ID: <490A101D.3030103@kylecordes.com> Kyle Cordes wrote: > Rather, it's much earlier. According to "cmp": > differ: byte 2098176010, line 131136001 The relevant part of the output file looks like the bit below. Look 4 lines in. See my previous message for an explanation of how this file is generated - the essence is that each line (with \n) is 16 bytes long, and its contents are the offset of its first byte. In a correct file, each number is 16 bigger than the next. 000002098175936 000002098175952 000002098175968 000002098175984 000002098225152 000002098225168 000002098225184 000002098225200 000002098225216 000002098225232 2098225152 - 2098175984 - 16 = 49152 UDPcast dropped 49152 bytes at this point. -- Kyle Cordes http://kylecordes.com From alain at knaff.lu Thu Oct 30 21:17:58 2008 From: alain at knaff.lu (Alain Knaff) Date: Thu, 30 Oct 2008 21:17:58 +0100 Subject: [Udpcast] More on the corrupt file problem In-Reply-To: <490A0B5B.2020903@kylecordes.com> References: <490A0B5B.2020903@kylecordes.com> Message-ID: <490A1676.2010108@knaff.lu> Kyle Cordes wrote: > To try to track down the udpcast corrupt file problem, I ran some more > tests. This time I used a ~50GB file, a sender, and only 1 receiver. > > side bytes > sender: 53687091200 > receiver: 53686091776 > > In all of my large-file runs, udp-receiver comes up a bit "short", it > missing some of the data, never "long". > > I created a 50 GB test file with predictable text data in it, suing this > ugly little program: > > #include > > // 16 bytes per entry. > // > int main(void) { > long long gb = 1024 * 1024 * 1024; > long long m = 50 * gb; > long long i; > for(i = 0; i printf("%.15lld\n", i); > } > } > > > so that I could easily look at the files. I found that the received file > ended with the same data as the sent file; in other words, the problem > is *not* a matter of terminating early, or other finishing-out process. > > Rather, it's much earlier. According to "cmp": > > differ: byte 2098176010, line 131136001 > > That's a little under 2 GB of the way in to a 50 GB file. > > Strangely, I ran repeated tests with 10 GB files, and didn't get any > corruption. > > > Alain - it would warm my heart to see you ack these messages, even if > you don't have a solution at hand. > I do get your messages, but for the moment I am somewhat busy on some other project (preparing the release of mtools version 4 with Unicode support). However, in some two weeks time I'll be more available to check out what is going on. The strange thing is, we do use udpcast for duplicating entire disks, most of which are larger than 50GB by now, and we never did notice any ill effect. A large piece of data missing in the middle would have been pretty obvious, but we've never have seen any of this so far. So apparently, it only happens under certain circumstances... and we need to understand what exactly these circumstances which are triggering this are. I appreciate your work on this subject, and I'm pretty confident that within a couple of more tests, you'll have identified what is going on (... making it easier for me to fix...) One suggestion (careful: this may take some time, and needs *huge* amounts of diskspace): try running udpcast under strace (strace -fo log.send udp-sender ... and strace -fo log.recv udp-receiver ...), and try to locate the system calls around the place where the missing data occurs (strace output should have reads and writes whose parameter is your textual data. The stretch of output between the reading or writing 000002098175984 and 000002098225152 is the interesting one here... Actually, to be precise, as udpcast reads and writes in largish chunks, you'll not see a read or write for every line. So the last read or write before the error will probably have a number less than 000002098175984, and the next write will have a number larger than 000002098225152, but you get the gist of it. Another weird thing is that although the problem happens relatively "early" in the file, it only occurs for certain minimum file sizes... just as if the file was being corrupted after the fact (say, after 10GB have been transferred.) It might be interesting to do a cmp midway through and see if the difference is already there "from the beginning..." (for instance, you may start your cmp as soon as your receive file reached size 2GB...) And, do several runs with the same input file always produce the error at the exact same spot? Regards, Alain From kyle at kylecordes.com Fri Oct 31 02:04:33 2008 From: kyle at kylecordes.com (Kyle Cordes) Date: Thu, 30 Oct 2008 20:04:33 -0500 Subject: [Udpcast] More on the corrupt file problem In-Reply-To: <490A1676.2010108@knaff.lu> References: <490A0B5B.2020903@kylecordes.com> <490A1676.2010108@knaff.lu> Message-ID: <490A59A1.7070702@kylecordes.com> Alain Knaff wrote: > support). However, in some two weeks time I'll be more available to > check out what is going on. Thanks for your reply. I will continue to investigate (such as with the ideas you describe below), and hopefully by the time you are able to attack it, I'll have enough diagnostic info to point to the problem. > The strange thing is, we do use udpcast for duplicating entire disks, > most of which are larger than 50GB by now, and we never did notice any I had assumed this was the case, which is why I found the corruption so surprising! (At the risk of sounding too critical, I was also surprised that udpcast doesn't do an end to end checksum or similar, it make me think of the oft-referenced 1981 paper: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf ) > One suggestion (careful: this may take some time, and needs *huge* > amounts of diskspace): try running udpcast under strace (strace -fo > log.send udp-sender ... and strace -fo log.recv udp-receiver ...), and I am not familiar with strace, but I will get familiar with it. I might combine this with a different idea I had: add a few lines of code to compare the file position with # of bytes udp-receiver thinks it wrong, and if they don't match, die. If I do this, it seems like the end of the strace would be at more or less exactly where the problem occurred. Do you have any feel for how much disk space I might need, to strace udp-receiver on a file of 50 GB? > Another weird thing is that although the problem happens relatively > "early" in the file, it only occurs for certain minimum file sizes... Yes, this is very weird. I will hopefully find a way to run the whole test in a loop - I have a couple of machines which could pound on it 24x7 for a few days. > just as if the file was being corrupted after the fact (say, after 10GB > have been transferred.) It might be interesting to do a cmp midway > through and see if the difference is already there "from the This seems unlikely to be at issue, since the trouble still occurs when I grab the output using: udp-receiver --pipe "tee somefile" >/dev/null Although my knowledge is incomplete, I don't think the OS will let udp-receiver reach through the pipe and "tee" to seek around on somefile. I also don't see how udp-receiver could possibly seek backward in to its output, because of this: $ grep seek *.c statistics.c: loff_t offset = lseek64(fd, 0, SEEK_CUR); statistics.c: off_t offset = lseek(fd, 0, SEEK_CUR); ... offhand I can't think of a way to move around in to a file without seek()ing. > And, do several runs with the same input file always produce the error > at the exact same spot? I will test this carefully, and report back. I think the answer is no, since in some test runs I udpcasted (do you mind your to three receivers, and each ends up with a different file length. -- Kyle Cordes http://kylecordes.com