Distributing Large Files - Implementation Part 1

This post is related to my previous post about distributing large files http://murzal-arsya.blogspot.com/2011/11/distributing-large-files.html.

Finally I implemented my distribution strategy as I explained in my previous post. Let me give a little bit info about the infrastructure.

1 Server:
Brand/Type: IBM System x3650 M3
OS: Centos 6
Service: Samba
NIC: 2 Gigabit

1 Switch: HP Procurve 1810 24g (Gigabit)

10 Clients:
Brand/Type: HP Pavilion HPE
OS: Windows 7
Apps: QGIS
NIC: 1 Gigabit


I have samba configured as shared storage. Within the share folder I put the large image files to be distributed to the clients. My strategy is creating a chain distribution, from the Server to PC-01, PC-01 to PC-02, and so on, until PC-10. I choose robocopy for its simplicity and it does the job well.
So, I created a batch file

Since I need to access the shared folder, I need to make sure that the connection could be established using correct user for authentication. In order to do that, before I run robocopy, I use net use command for authenticating.
>net use \\SERVER\share_images /USER:user1 password1
then I could execute robocopy with something like the following:
>robocopy /E /PURGE /COPY:DAT \\SERVER\share_images C:\images

The most ideal way to trigger the synchronization is when there is change in each share folder, and queuing the sync processes when there are multiple changes detected. It is beyond the scope of this post.

Distributing Large Files

One of the requirements of my new project is to be able to distribute large files to 10-15 clients as fast as possible. Fortunately, all (server and clients) NICs supports Gigabit connectivity.
I tested it first simply by copying large data from server to many clients simultaneously. Of course, it was a disaster. While theoretically the copying speed would decrease proportional to the number of clients, in reality the network connection between clients and server, and also between clients got disrupted. Network was down for sometimes. I did not make any further investigation on what was happening, all I know is this is far from the desired speed even if it was going well.
What's next? I gather some info's on how to distribute large files over LAN. Some suggest bittorrent, parallel filesystem, multicast copy, (win) mesh, and some other way. My concern is that I have to use a solution which is as simple as possible, since there would not be any expert IT guy around in the future.
While bittorrent sounds pretty promising, I had to leave out this option. Not only because of the capability of the human resources that will maintain the system, also it would take some times to prepare the large file to be distributed by bittorrent. Another thing is that small torrent packets could really saturate the network.

About parallel filesystem, I have to ditch this one too, because this would be more complicated and confusing for the basic admin guy. Also I need to mention that all clients are Windows 7, not really sure there are stable parallel filesystem support

Multicast copy should be interesting, but could not find any apps for Windows 7.

So, I figured that I need to make my own simple solution. The goal here is to distribute large files, around 1-5 GigaBytes each, without overloading the connection between source (server) and target (pc).

I simply came up with creating some kind of chain distribution. Transfer file from Server to PC1, and from PC1 to PC2, and so on. Doing this, I could use all of the available incoming gigabit bandwidth as well as the outgoing one. I tested it and worked quite well. Of course there will be a problem if one of the PC is down, then the distribution could not be continued to the rest of the PCs after that. So, rerouting must be done either manually or automatically.

As for the tool to copy the data, one can use simple copy, robocopy, synctoy, and other mechanism. I am planning to use command line robocopy.

I also plan to use load balancing technique using both NICs on the server. So, I can create 2 chains within the network, and perform the transfer simultaneously to those chains.

Broadcast Ping

Broadcast Ping

Perform the following commands:

$ ifconfig eth0 | grep -i bcast
         inet addr:10.10.44.44  Bcast:10.10.255.255  Mask:255.255.0.0

$ ping -b 10.10.255.255

You should see several IP addresses respond to ping.

Problem with Java SocketChannel

Problem with Java SocketChannel

I created a simple java network application and wanted to put it as a service (windows and linux). To do that, I had to implement shutdown mechanism that releases all resources in proper manner for my java app. Obviously, I used Socket (and ServerSocket) class, which runs in separate thread, and used InputStream and OutputStream to read and write data.
So, the problem comes when I need to stop the thread that has Socket which block for incoming data (by invoking read method for example). Clearly, I need to be able to interrupt that blocking point and exit the thread gracefully.
After searching a bit in the internet, I found out about SocketChannel, which implements InterruptibleChannel. This is good, so I could simply send interrupt signal to the thread and the SocketChannel will also get it and quit from blocking state.

But there is a problem (perhaps bug) with the implementation of SocketChannel, that would prevent concurrent read and write to it. Please refer to sun java bug id 4774871 and 4509080.

The work around is like the following (taken from bug #4774871 details):

CUSTOMER WORKAROUND :
The above program works if you change the wrapChannel() method to wrap the SocketChannel to a dummy channel that only forwards the calls to it. It works because it is not an instance of SelectableChannel:

private static ByteChannel wrapChannel(final ByteChannel channel) {

   return new ByteChannel() {
      public int write(ByteBuffer src) throws IOException {
         return channel.write(src);
      }

      public int read(ByteBuffer dst) throws IOException {
         return channel.read(dst);
      }

      public boolean isOpen() {
         return channel.isOpen();
      }

      public void close() throws IOException {
         channel.close();
      }
   };
}

ssh tunneling

ssh tunneling

In the client execute:

$ ssh -N -D user@CompB

where CompB is the server

To use this ssh tunneling with internet browser, for example firefox, change the "Connection Settings", under "Manual proxy configuration", update the "SOCKS Host:" to localhost:any_free_port. Notice that any_free_port here must be the same one as the one used in ssh.

Opera browser does not have socks proxy configuration. One way to do it is by using "tsocks."
Install tsocks:

$ sudo apt-get install tsocks

edit /etc/tsocks.conf, and change
server = 127.0.0.1 # this is localhost
server_port = # again, this must match with the port used in ssh.