From vir.iratus1686 at gmail.com Tue Apr 20 23:48:14 2010 From: vir.iratus1686 at gmail.com (Michael Gunter) Date: Tue, 20 Apr 2010 11:48:14 -1000 Subject: [Udpcast] Jumbo Frames in UDPcast? Message-ID: I'm a student and work at a school where we image our machines using UDPcast. It has been a great tool for installing entire classrooms quickly, but we recently upgraded all of our hardware and are now running gigabit ethernet. We've configured the eth0 to have an MTU of 9000 with the following line ifconfig eth0 mtu 9000 on both the sending and receiving computers. After preparing the machines we use the udp-sender command to copy the master hard drive over the network and include the switch --blocksize=9000 but it doesn't seem to affect our transfer speeds at all. Without changing the MTU or blocksize it sends at around 180 Mbps and after changing the MTU and blocksize it still sends at about the same speed. Is there possibly another bottleneck we've overlooked such as bus speeds or hard drive transfer speeds? Or does UDPcast not support jumbo frames? I saw somewhere in the archives that the blocksize can be reduced if your network doesn't support 1500 byte frames, but never anything about increasing the size. Please advise, thank you very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikes at kuentos.guam.net Wed Apr 21 04:07:21 2010 From: mikes at kuentos.guam.net (Michael D. Setzer II) Date: Wed, 21 Apr 2010 12:07:21 +1000 Subject: [Udpcast] Jumbo Frames in UDPcast? In-Reply-To: References: Message-ID: <4BCE5DD9.9858.112075@mikes.kuentos.guam.net> What are the true speeds of the hard disks? You are only looking at the network speed, but the data also needs to be written to the disk, and disk imaging does massive disk access, so the hard disk buffer is quickly used up. If you can, run hdparm -Tt /dev/sda (or whatever the disk is) That will report to you the disk speed using the buffer and not. On the machine I am using at the moment. /dev/sda: Timing cached reads: 4898 MB in 2.00 seconds = 2450.11 MB/sec Timing buffered disk reads: 254 MB in 3.01 seconds = 84.46 MB/sec In the past, I had show students how using 100MB nics we could get an over speed of about 30MB/sec. After putting 1GB nics in two machines, the same transfer gave about 65MB/sec. The nic was 10 times faster, but now the hard disk was the limiting factor. On 20 Apr 2010 at 11:48, Michael Gunter wrote: Date sent: Tue, 20 Apr 2010 11:48:14 -1000 From: Michael Gunter To: udpcast at udpcast.linux.lu Subject: [Udpcast] Jumbo Frames in UDPcast? > > I'm a student and work at a school where we image our machines using UDPcast. It has been a > great tool for installing entire classrooms quickly, but we recently upgraded all of our hardware > and are now running gigabit ethernet. We've configured the eth0 to have an MTU of 9000 with > the following line > > ifconfig eth0 mtu 9000 > > on both the sending and receiving computers. After preparing the machines we use the udp- > sender command to copy the master hard drive over the network and include the switch > > --blocksize=9000 > > but it doesn't seem to affect our transfer speeds at all. Without changing the MTU or blocksize it > sends at around 180 Mbps and after changing the MTU and blocksize it still sends at about the > same speed. Is there possibly another bottleneck we've overlooked such as bus speeds or hard > drive transfer speeds? Or does UDPcast not support jumbo frames? I saw somewhere in the > archives that the blocksize can be reduced if your network doesn't support 1500 byte frames, > but never anything about increasing the size. Please advise, thank you very much! > +----------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mikes at kuentos.guam.net mailto:msetzerii at gmail.com http://www.guam.net/home/mikes Guam - Where America's Day Begins +----------------------------------------------------------+ http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489) BOINC at HOME CREDITS SETI 9612639.793425 | EINSTEIN 3941533.260851 ROSETTA 1882529.685530 | ABC 728490.575656 From AYChen at Avinta.com Wed Apr 21 04:05:17 2010 From: AYChen at Avinta.com (Abraham Y. Chen) Date: Tue, 20 Apr 2010 19:05:17 -0700 Subject: [Udpcast] Source for Windows 201004201849.AYC In-Reply-To: <363A11502F1B45418925C49FDDB72F2416E9413DB8@MAIL-LQ.baylor.edu> References: <363A11502F1B45418925C49FDDB72F2416E9413DB8@MAIL-LQ.baylor.edu> Message-ID: <4BCE5D5D.8000801@Avinta.com> Hi, Michael: I, too, have been hoping to develop a generic networking monitoring utility for both Linux and Windows machines by utilizing a part of the UDPcast capabilities. Being able to generate both executable codes by compiling from the same source code is a definite encouragement to my plan. Reviewing the MSYS that you have used, it seems to me that it needs to be under some kind of software shell environment to work? Did you start with a PC with Windows as the OS? Could you please outline your setup, so that I can duplicate your success? Thanks very much in advance. Abe (2010-04-20, 19:04) Website: www.Avinta.com Holdridge, Michael B wrote: > Great, it compiled! Thanks! Now I just have to edit the code without breaking it. Hopefully Java and C++ are similar enough that it take me too long to figure it out. > > From: udpcast-bounces at udpcast.linux.lu [mailto:udpcast-bounces at udpcast.linux.lu] On Behalf Of HouSoft > Sent: Sunday, March 21, 2010 4:26 AM > To: udpcast at udpcast.linux.lu > Subject: [Udpcast] Source for Windows > > using mysys to compile the source files,it's easy! > -------------- > http://msys-cn.googlecode.com/files/MSYS-Update.rar > ------------- > ./configure > make > > > ------------------ Original ------------------ > From:"Holdridge, Michael B"; > Date:2010年3月21日(星期天) 中午1:41 > To:"udpcast at udpcast.linux.lu"; > Subject:[Udpcast] Source for Windows > > Is the source code supposed to compile in Windows without modification or would I have to port the code? Or are there sources for Windows available? I have tried using Dev C++ to compile the source files but am not having any luck. I’m fairly new to Linux and and have little experience with C, so I am not sure where to start. I know there are Windows binaries to download but this is not what I am looking for. I have been asked for a solution to distribute large game analysis videos to multiple workstations simultaneously. Udpcast does this perfectly, but I would like to make it more end user friendly. Any help would be greatly appreciated. > > Thanks, > Michael Holdridge > > _______________________________________________ > Udpcast mailing list > Udpcast at udpcast.linux.lu > https://udpcast.linux.lu/cgi-bin/mailman/listinfo/udpcast > From alain at knaff.lu Wed Apr 21 10:21:36 2010 From: alain at knaff.lu (Alain Knaff) Date: Wed, 21 Apr 2010 10:21:36 +0200 Subject: [Udpcast] Jumbo Frames in UDPcast? In-Reply-To: References: Message-ID: <4BCEB590.1010107@knaff.lu> On 20/04/10 23:48, Michael Gunter wrote: > I'm a student and work at a school where we image our machines using > UDPcast. It has been a great tool for installing entire classrooms quickly, > but we recently upgraded all of our hardware and are now running gigabit > ethernet. We've configured the eth0 to have an MTU of 9000 with the > following line > > ifconfig eth0 mtu 9000 > > on both the sending and receiving computers. After preparing the machines we > use the udp-sender command to copy the master hard drive over the network > and include the switch > > --blocksize=9000 Just one remark: each frame has a certain amount of "overhead" (Udpcast headers, IP headers, Ethernet headers), so in order to avoid fragmentation (and performance loss), you need to stay slightly under your MTU. So, for example: --blocksize 8952 > > but it doesn't seem to affect our transfer speeds at all. Without changing > the MTU or blocksize it sends at around 180 Mbps and after changing the MTU > and blocksize it still sends at about the same speed. Is there possibly > another bottleneck we've overlooked such as bus speeds or hard drive > transfer speeds? Or does UDPcast not support jumbo frames? I saw somewhere > in the archives that the blocksize can be reduced if your network doesn't > support 1500 byte frames, but never anything about increasing the size. > Please advise, thank you very much! > > > > > _______________________________________________ > Udpcast mailing list > Udpcast at udpcast.linux.lu > https://udpcast.linux.lu/cgi-bin/mailman/listinfo/udpcast Regards, Alain From vir.iratus1686 at gmail.com Wed Apr 21 13:12:01 2010 From: vir.iratus1686 at gmail.com (Michael Gunter) Date: Wed, 21 Apr 2010 01:12:01 -1000 Subject: [Udpcast] Fwd: Jumbo Frames in UDPcast? In-Reply-To: References: <4BCEB590.1010107@knaff.lu> Message-ID: ---------- Forwarded message ---------- From: Michael Gunter Date: Wed, Apr 21, 2010 at 1:11 AM Subject: Re: [Udpcast] Jumbo Frames in UDPcast? To: Alain Knaff After some further testing, it seems that our bottleneck is in the compression and writing to the hard drive. To test, we set it up so we were transferring /dev/zero through the nic into the /dev/null of another machine. This revealed transfer speeds reaching close to 1 GB a second, but of course, nothing was being read from the hard drive, compressed, or written to a hard drive. We also did a test transferring a file without lzop compression (which I forgot to mention we were using originally with an MTU of 1500) and it did transfer at speeds almost double our normal transfer speeds, so around 400 MB a second. The only problem is, sending the uncompressed data at twice the speed took pretty much the same amount of time as sending it compressed in smaller frames. We also took into account the possibility of overhead and lowered the --blocksize to a safe 8500. I have been looking into possibly combining GSO (Generic Segmentation Offloading) with UDPcast to try and remove some of the overhead on the processor, but have yet to actually try it yet. Does anyone know if it would effectively increase the transfer speed or if it can even be done? Thank you very much for your responses thus far. On Tue, Apr 20, 2010 at 10:21 PM, Alain Knaff wrote: > On 20/04/10 23:48, Michael Gunter wrote: > > I'm a student and work at a school where we image our machines using > > UDPcast. It has been a great tool for installing entire classrooms > quickly, > > but we recently upgraded all of our hardware and are now running gigabit > > ethernet. We've configured the eth0 to have an MTU of 9000 with the > > following line > > > > ifconfig eth0 mtu 9000 > > > > on both the sending and receiving computers. After preparing the machines > we > > use the udp-sender command to copy the master hard drive over the network > > and include the switch > > > > --blocksize=9000 > > Just one remark: each frame has a certain amount of "overhead" (Udpcast > headers, IP headers, Ethernet headers), so in order to avoid > fragmentation (and performance loss), you need to stay slightly under > your MTU. > > So, for example: > --blocksize 8952 > > > > > but it doesn't seem to affect our transfer speeds at all. Without > changing > > the MTU or blocksize it sends at around 180 Mbps and after changing the > MTU > > and blocksize it still sends at about the same speed. Is there possibly > > another bottleneck we've overlooked such as bus speeds or hard drive > > transfer speeds? Or does UDPcast not support jumbo frames? I saw > somewhere > > in the archives that the blocksize can be reduced if your network doesn't > > support 1500 byte frames, but never anything about increasing the size. > > Please advise, thank you very much! > > > > > > > > > > _______________________________________________ > > Udpcast mailing list > > Udpcast at udpcast.linux.lu > > https://udpcast.linux.lu/cgi-bin/mailman/listinfo/udpcast > > Regards, > > Alain > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael_Holdridge at baylor.edu Thu Apr 22 07:06:55 2010 From: Michael_Holdridge at baylor.edu (Holdridge, Michael B) Date: Thu, 22 Apr 2010 00:06:55 -0500 Subject: [Udpcast] Source for Windows 201004201849.AYC In-Reply-To: <4BCE5D5D.8000801@Avinta.com> References: <363A11502F1B45418925C49FDDB72F2416E9413DB8@MAIL-LQ.baylor.edu> <4BCE5D5D.8000801@Avinta.com> Message-ID: <363A11502F1B45418925C49FDDB72F2416EA6626BF@MAIL-LQ.baylor.edu> Hi Abe, You're right, I think, about the shell environment. I am no expert (and anyone please correct me if make an incorrect statement), but it seems that MSYS provides a Linux/Unix-like environment and command shell. It also includes minGW, a GCC-like compiler with support for the Windows API, which you can run from the MSYS shell. I am running Windows XP. It was pretty simple and I am relatively new to Linux. Simply download the RAR extract where ever you like and run msys.bat. Once you are in the command shell use the normal commands like "configure", "make", etc. that you would use in a Linux environment. As a side note, I did manage to get Dev-C++ to work after I was successful compiling in MSYS. I don't think the Dev-C++ success is related to having MSYS installed. Dev-C++ includes its own minGW as the backend compiler but is a Windows-like IDE. I believe the problem was that when compiling, it would create a new file called makefile.win. For whatever reason, it would not compile using this makefile. So you must go to "Project Options" and the "Makefile" tab, check "Use custom makefile" and select the correct file which should simply be called "makefile". After doing that, Dev-C++ would correctly compile udpcast without errors. Hope this helps, Michael -----Original Message----- From: Abraham Y. Chen [mailto:AYChen at Avinta.com] Sent: Tuesday, April 20, 2010 9:05 PM To: Holdridge, Michael B Cc: udpcast at udpcast.linux.lu Subject: Re: [Udpcast] Source for Windows 201004201849.AYC Hi, Michael: I, too, have been hoping to develop a generic networking monitoring utility for both Linux and Windows machines by utilizing a part of the UDPcast capabilities. Being able to generate both executable codes by compiling from the same source code is a definite encouragement to my plan. Reviewing the MSYS that you have used, it seems to me that it needs to be under some kind of software shell environment to work? Did you start with a PC with Windows as the OS? Could you please outline your setup, so that I can duplicate your success? Thanks very much in advance. Abe (2010-04-20, 19:04) Website: www.Avinta.com Holdridge, Michael B wrote: > Great, it compiled! Thanks! Now I just have to edit the code without breaking it. Hopefully Java and C++ are similar enough that it take me too long to figure it out. > > From: udpcast-bounces at udpcast.linux.lu [mailto:udpcast-bounces at udpcast.linux.lu] On Behalf Of HouSoft > Sent: Sunday, March 21, 2010 4:26 AM > To: udpcast at udpcast.linux.lu > Subject: [Udpcast] Source for Windows > > using mysys to compile the source files,it's easy! > -------------- > http://msys-cn.googlecode.com/files/MSYS-Update.rar > ------------- > ./configure > make > > > ------------------ Original ------------------ > From:"Holdridge, Michael B"; > Date:2010年3月21日(星期天) 中午1:41 > To:"udpcast at udpcast.linux.lu"; > Subject:[Udpcast] Source for Windows > > Is the source code supposed to compile in Windows without modification or would I have to port the code? Or are there sources for Windows available? I have tried using Dev C++ to compile the source files but am not having any luck. I’m fairly new to Linux and and have little experience with C, so I am not sure where to start. I know there are Windows binaries to download but this is not what I am looking for. I have been asked for a solution to distribute large game analysis videos to multiple workstations simultaneously. Udpcast does this perfectly, but I would like to make it more end user friendly. Any help would be greatly appreciated. > > Thanks, > Michael Holdridge > > _______________________________________________ > Udpcast mailing list > Udpcast at udpcast.linux.lu > https://udpcast.linux.lu/cgi-bin/mailman/listinfo/udpcast > From hou at yryz.net Sat Apr 24 13:58:10 2010 From: hou at yryz.net (HouSoft) Date: Sat, 24 Apr 2010 19:58:10 +0800 Subject: [Udpcast] About Udpcast FLAG_SN Message-ID: Hi AiLan: Why use FLAG_SN, do not DSC_REDUCING or DSC_REDUCING the slice of size? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From alain at knaff.lu Sat Apr 24 15:30:15 2010 From: alain at knaff.lu (Alain Knaff) Date: Sat, 24 Apr 2010 15:30:15 +0200 Subject: [Udpcast] About Udpcast FLAG_SN In-Reply-To: References: Message-ID: <4BD2F267.8030905@knaff.lu> HouSoft wrote: > Hi AiLan: > > Why use FLAG_SN, do not DSC_REDUCING or DSC_REDUCING the slice of size? > > Thanks! FLAG_SN (--full-duplex) means that the network supports full duplex, i.e. that acknowledgment data sent back from the receivers does not disturb new data sent from the sender to the receiver. This is the case with most networks today (when using a switch rather than a hub). Updcast can take advantage of this by starting to send the next slice before all acknowledgments of the previous one have been received. DSC_REDUCING is used by the algorithm to find the optimal slice size. This algorithm is only used on half-duplex networks, and works by starting at a certain size, than keeping to add 25% slice size until transmission errors start showing up, and then reducing until transmission errors are gone. Such errors due to large slices can occur under certain timing conditions on half-duplex networks, and thus this algorithm is not needed in full duplex. Moreover, in full duplex mode, the slice size has no impact on performance, as the next slice can be transmitted right away, without needing to wait for the acknowledgments of the previous once. Regards, Alain From hou at yryz.net Sat Apr 24 18:20:24 2010 From: hou at yryz.net (=?gbk?B?0rvIy9PO198=?=) Date: Sun, 25 Apr 2010 00:20:24 +0800 Subject: [Udpcast] About Udpcast FLAG_SN Message-ID: I tried to convert it to object-oriented (Delphi), Sender has been completed. Good results, the performance did not decrease, the structure is more clear. Also deepened my understanding of it! Thanks Alain! ------------------ Original ------------------ From: "Alain Knaff"; Date: 2010年4月24日(星期六) 晚上9:30 To: "HouSoft"; Subject: Re: [Udpcast] About Udpcast FLAG_SN HouSoft wrote: > Hi AiLan: > > Why use FLAG_SN, do not DSC_REDUCING or DSC_REDUCING the slice of size? > > Thanks! FLAG_SN (--full-duplex) means that the network supports full duplex, i.e. that acknowledgment data sent back from the receivers does not disturb new data sent from the sender to the receiver. This is the case with most networks today (when using a switch rather than a hub). Updcast can take advantage of this by starting to send the next slice before all acknowledgments of the previous one have been received. DSC_REDUCING is used by the algorithm to find the optimal slice size. This algorithm is only used on half-duplex networks, and works by starting at a certain size, than keeping to add 25% slice size until transmission errors start showing up, and then reducing until transmission errors are gone. Such errors due to large slices can occur under certain timing conditions on half-duplex networks, and thus this algorithm is not needed in full duplex. Moreover, in full duplex mode, the slice size has no impact on performance, as the next slice can be transmitted right away, without needing to wait for the acknowledgments of the previous once. Regards, Alain -------------- next part -------------- An HTML attachment was scrubbed... URL: