Distributing Large Files - Implementation Part 1

This post is related to my previous post about distributing large files http://murzal-arsya.blogspot.com/2011/11/distributing-large-files.html.

Finally I implemented my distribution strategy as I explained in my previous post. Let me give a little bit info about the infrastructure.

1 Server:
Brand/Type: IBM System x3650 M3
OS: Centos 6
Service: Samba
NIC: 2 Gigabit

1 Switch: HP Procurve 1810 24g (Gigabit)

10 Clients:
Brand/Type: HP Pavilion HPE
OS: Windows 7
Apps: QGIS
NIC: 1 Gigabit


I have samba configured as shared storage. Within the share folder I put the large image files to be distributed to the clients. My strategy is creating a chain distribution, from the Server to PC-01, PC-01 to PC-02, and so on, until PC-10. I choose robocopy for its simplicity and it does the job well.
So, I created a batch file

Since I need to access the shared folder, I need to make sure that the connection could be established using correct user for authentication. In order to do that, before I run robocopy, I use net use command for authenticating.
>net use \\SERVER\share_images /USER:user1 password1
then I could execute robocopy with something like the following:
>robocopy /E /PURGE /COPY:DAT \\SERVER\share_images C:\images

The most ideal way to trigger the synchronization is when there is change in each share folder, and queuing the sync processes when there are multiple changes detected. It is beyond the scope of this post.

Distributing Large Files

One of the requirements of my new project is to be able to distribute large files to 10-15 clients as fast as possible. Fortunately, all (server and clients) NICs supports Gigabit connectivity.
I tested it first simply by copying large data from server to many clients simultaneously. Of course, it was a disaster. While theoretically the copying speed would decrease proportional to the number of clients, in reality the network connection between clients and server, and also between clients got disrupted. Network was down for sometimes. I did not make any further investigation on what was happening, all I know is this is far from the desired speed even if it was going well.
What's next? I gather some info's on how to distribute large files over LAN. Some suggest bittorrent, parallel filesystem, multicast copy, (win) mesh, and some other way. My concern is that I have to use a solution which is as simple as possible, since there would not be any expert IT guy around in the future.
While bittorrent sounds pretty promising, I had to leave out this option. Not only because of the capability of the human resources that will maintain the system, also it would take some times to prepare the large file to be distributed by bittorrent. Another thing is that small torrent packets could really saturate the network.

About parallel filesystem, I have to ditch this one too, because this would be more complicated and confusing for the basic admin guy. Also I need to mention that all clients are Windows 7, not really sure there are stable parallel filesystem support

Multicast copy should be interesting, but could not find any apps for Windows 7.

So, I figured that I need to make my own simple solution. The goal here is to distribute large files, around 1-5 GigaBytes each, without overloading the connection between source (server) and target (pc).

I simply came up with creating some kind of chain distribution. Transfer file from Server to PC1, and from PC1 to PC2, and so on. Doing this, I could use all of the available incoming gigabit bandwidth as well as the outgoing one. I tested it and worked quite well. Of course there will be a problem if one of the PC is down, then the distribution could not be continued to the rest of the PCs after that. So, rerouting must be done either manually or automatically.

As for the tool to copy the data, one can use simple copy, robocopy, synctoy, and other mechanism. I am planning to use command line robocopy.

I also plan to use load balancing technique using both NICs on the server. So, I can create 2 chains within the network, and perform the transfer simultaneously to those chains.

Using django-admin.py within cygwin

Sometimes I still work in windows environment. I always have cygwin installed in windows environment since I use a lot of *nix tools and scripts.
Currently, I am working a lot with python and django, very nice scripting language and powerful web framework. I used native windows python installation, instead of installing it within cygwin.
The problem occur when I tried to invoke django-admin.py script in cygwin's bash shell. It won't run. It is due to windows version of python expecting windows path of the django-admin.py script as the argument.
Here is the exact situation. Windows python expecting, something like this:

python c:\Python26\Scripts\django-admin.py

but since I run that command within cygwin, it will become like the following:

python /cygdrive/c/Python26/Scripts/django-admin.py

and throws some error messages.

So, I searched a bit and found out that cygwin has a tool that converts *nix path to windows path. The name of the tool is cygpath.

And here is what I did to solve the problem.
I created the following file:
/usr/local/bin/django-admin.py

which contains:
python.exe "$(cygpath -aw "/cygdrive/c/Python26/Scripts/django-admin.py")"

Of course you need to make /usr/local/bin/django-admin.py executable

chmod +x /usr/local/bin/django-admin.py

and you can run django-admin.py from anywhere.


Check out the following links for more information:
http://www.cygwin.com/cygwin-ug-net/using-effectively.html
http://www.cygwin.com/cygwin-ug-net/using-utils.html#cygpath

ACT!2009 SDK

ACT!2009 SDK

Nice thing about the recent version of ACT is that they provide a way to extend the functionality using either C# or VB.NET.

You can either create a plugin (dll) to extend the existing functionality of ACT, or create a separate stand alone application that uses the ACT Framework.

I haven't done much with the SDK, still trying to get a feel with the whole Framework thing. Besides, I am also not so sure it's a good idea to create many plugins and applications, since there are some nasty bugs in ACT!2009, like I mentioned in my earlier post.

In order to use the SDK, you need to have the original Installer CD. Go to the following folder to find all necessary dll's ACT!2009\PremiumEX\ACTWG\GlobalAssemblyCache. This the case where you don't have the ACT!2009 installed in your system.
If you have ACT installed in your system, you can go to C:\WINDOWS\assembly\GAC_MSIL and find all the required dll's. Notice that you can not see GAC_MSIL folder from explorer, but instead you can see it from CLI.

ACT!2009 Backup

ACT!2009 Backup

Do you backup your ACT!2009 DB regularly? If not, you should configure automatic backup.
Use ACT Scheduler to schedule automatic backup. Refer to this link on how to set it up.

I really suggest to store the backup in different place then where the ACT!2009 data is located. You might face some problems if you put in the default backup folder. Refer to this link for more information about this.

If somehow you decided to store the backup in other computer/server that needed login validation, you need to configure the ACT Scheduler Service to use proper login account. Click here to see how to do it.

List of problem with ACT!2009 + Outlook 2007

List of problem with ACT!2009 + Outlook 2007

List of ACT!2009 and Outlook 2007 known problem (so far) as per 20090221

1. After performing lookup by example in ACT, the information shown in the fields are from the previous contact. This can be quite confusing for user. Until now, there is no solution for this issue. The only thing we can do is, pressing "F5", to refresh the page, after performing any lookup.

2. Several fields are not shown in import/export wizard in ACT. I consider this issue as the worst one. I was facing a lot of trouble when trying to import some contacts from other database. Too many missing data, which really makes the import/export process is useless. True, we can always use the quick export to excel, but the number of the fields is then limited to what is supported by excel. No good.
Click here to get more information about the problem (but not the solution).

3. Again in ACT. If you happened to have date or annual event field type in your layout, be very careful. Usually, after I make any change to the date field, I click outside, such as the layout background, so when I use the scroll wheel in my mouse, the date will not get changed. Surprisingly, the date still get changed. So, either use the scroll bar, or click on text field then use the scroll wheel.

4. When performing mail merge to a lot of contacts, for example a thousand, outlook crashes. Not good. Solution? Use different email client that can handle big number of emails.

5. I created a simple vba script to perform mail merge from outlook. I was hoping that using vba script, I could perform the mail merge without outlook crashing, and the email would be attached in history. Emails went through just fine, but this time ACT.Outlook.Service.exe crashed.

ACT + Outlook 2007 + PST over the LAN = Huge Mess

ACT + Outlook 2007 + PST over the LAN = Huge Mess

I had some problems with ACT!2009 and Outlook 2007. From Outlook, I kept getting pop-up error message saying that the pst file was locked, probably used in some other terminal. ACT!2009 kept giving me pop-up error message saying something like Act! Value cannot be null. Parameter name:pUnk. What an error message, and who is this pUnk guy, anyway.

So, initially the pst file was stored in the server, so different user can take care the incoming file at a time. True, that Exchange server should be used in this particular case, but somehow they don't want to use it, with many different reason, which I don't agree. The user insisted that they want to have convenience of not having to switch back and forth using different computer. Therefore, pst file stored in server.

When outlook showed error saying that the pst file was locked and being used in other terminal, one thing I had in my mind was, probably that's true. So, I made sure no one else is using it. Still, the error occurred many times.
The error message from ACT was really gave me no clue on what's going on.

After spending a lot of time searching here and there, I came across the following information: Personal folder files are unsupported over a LAN or over a WAN link, http://support.microsoft.com/kb/297019

My intuition told me that this must have something to do with the problems we were having. Well, it turned out to be that storing pst file in the server and accessed over the network is a terrible idea.

Once I convinced users that since they don't have Exchange server and pst file is not designed to be used over network, they agree to move the pst files locally. Problem solved.

ACT!2000 to ACT!2009 Migration

ACT!2000 to ACT!2009 Migration

Currently, I work a lot with ACT!2009, Contact and Customer Relationship Management Software. This is pretty new thing for me. Interesting and troublesome at the same time.

I had to migrate all data from ACT!2000 to ACT!2009. According to the ACT Knowledge Base, the procedure should be simple, and straight forward. The only problem is, the database in ACT!2000 was corrupted, quite badly. ACT diagnostic tool could not fix the problem, and the only way to fix this is by using import/export features. Please go here to get detailed information on how to use import and export to repair ACT!2000 database.

The process was time consuming and when there were duplicate contact records found during import, user interaction was needed. Apparently the there were a lot of duplicate contact found in the database.
One lesson learned here, which suppose to be done by previous user, is to resolve duplicate contact record.

Once the repair process done, I simply opened the previous version of ACT database in the new ACT, and the conversion process begins.
After the conversion done, there are many fields that are not converted to the properly type. For example, date field become annual event. Memo files become regular text fields. No good.

ACT Knowledge Base is a place to go for ACT users.

Resolve Windows hostname in Linux

Resolve Windows hostname in Linux

I work a lot in heterogeneous environment, and ofcourse, windows is one of the system in that network. So, I would like to be able to resolve windows hostname (netbios) and access it's resource. In order to do that, there are packages needed to be installed first.

$ sudo apt-get install winbind

after that, edit /etc/nsswitch.conf configuration file.
Search for word "hosts", and add "wins" at the end of the line. Below is an example of "hosts" configuration after update.

# /etc/nsswitch.conf
#
# ....

hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4 wins

#....

after that, restart winbind service (if necessary). Now, windows computers should be able to be accessed using it's windows hostname. Pretty convenient way to access those windows computer in the network.

If you are still not able to resolve windows hostname, perhaps there is a firewall running in your computer. In that case you need to add extra rule to allow udp.

I use firestarter for my personal firewall, and what I did was edit the /etc/firestarter/inbound/setup configuration file.
Search for:
$IPT -A INBOUND -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
and put the following rule right after it:
$IPT -A INBOUND -s $NET -p udp -m state --state NEW -j ACCEPT

restart firestarter, to load the new config. Try to access windows computer using it's hostname again.


I just found out a way to configure UFW for this matter.
First edit: /etc/default/ufw
Search for:

IPT_MODULES="nf_conntrack_ftp nf_nat_ftp nf_conntrack_irc nf_nat_irc"

and change it to:
IPT_MODULES="nf_conntrack_ftp nf_nat_ftp nf_conntrack_irc nf_nat_irc nf_conntrack_netbios_ns"


Then: reload ufw
# sudo ufw reload

Next step: Configure UFW Rules
Allow outgoing UDP to port: 137, 138
Allow outgoing TCP to port: 139, 445