Advertise SFTP on Network using Avahi

I just realized that I can broadcast / advertise my SFTP server to other computers connected to the network.

Go to /etc/avahi/services
and create a file named sftp.service, which contains:





   %h SFTP
 
   
      _sftp-ssh._tcp
      22
 

Now, you can see your SFTP server in Nautilus.

This guy has more complete list on what can be broadcast using avah i

Django Login and Sign-Up Page Together

It is very common to combine login and sign-up form in one page, so I implemented it in django. Here's how I do it.

First, create UserForm class which contains whatever information you need, such as: username, password, full name, address, email, gender, etc.

Second, add login url to your urls.py, something like the following:

(r'^/login/$', 'django.contrib.auth.views.login', { 'template_name': 'myapp/login.html', 'reg_form': UserForm() }),

Third, create in your template folder myapp/login.html which content is like shown in docs.djangoproject.com/en/dev/topics/auth/ (search for sample registration/login.html)

Make sure you use the exact same variable name (form, next, etc), unless you know what you are doing.

Also, in this template, add another html form and render your UserForm object (in my case the variable name is reg_form.

Fourth, create sign_up view, which perform whatever it is necessary in order to save the new user. Make sure you add { 'form': django.contrib.auth.forms.AuthenticationForm() } to the context data.

Sixth, add sign_up url to your urls.py, something like the following:

(r'^/sign-up/$', myapp.views.sign_up ),

Notes:

I created my own User class with additional information that django user (auth_user) does not provide. In order to do that, I create (custom) User with a OneToOne relationship with django user which is also the primary_key.

That's how I do it. Hopefully, give you some ideas if you want to implement something similar.

Consuming Django Tastypie RESTful WS with Apache HTTPClient and GSON

Creating RESTful Web Service is so simple using Django Tastypie. It's pretty much just about registering which model you want to expose. Even more amazing is that filtering feature. Read here to get more info about it.

I decided to create Java frontend to replace the existing, which uses Python Gtk. To keep with the existing look and feel I use java-gnome binding. The transition from python gtk to java-gnome is quite smooth. Java-gnome has such a good design, so the only thing that I need to for the transition is just adapting to the java paradigm, which is wonderful.

To get the JSON object from the web service, I chose Apache HTPClient, just to make things simple. I know that there are a lot of java REST client, such as restlet, jersey, etc. I stick with the simplest one.
The next thing is to figure out how to transform JSON result from the RESTful web service to POJO.  Here comes GSON to the rescue. I did not want to use python attribute name convention in my java codes. So, this one I need to solve. Fortunately, GSON comes with clever and handy feature named FieldNamingPolicy. Obviously they have really good experience in different programming language hence this feature available.

To give example on how to use it, taken from the site:


SomeObject someObject = new SomeObject("first", "second");
Gson gson = new GsonBuilder().setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES).create();
String jsonRepresentation = gson.toJson(someObject);
System.out.println(jsonRepresentation);


Awesome! That simple and you can keep the java attribute naming convention and make your java codes look good.

Debian with Ubuntu Fonts

Here is how I configure my Debian system to use the same font config as Ubuntu.
Simply download, fontconfig-config_2.8.0-3ubuntu9_all.deb. The link will take you to Ubuntu 12.04 fontconfig-config deb package.
After you saved that deb file, extract it to get its contents:
$ dpkg-deb -x fontconfig-config_2.8.0-3ubuntu9_all.deb /tmp/extract/

then replace the content of Debian's /etc/fonts/* with data from /tmp/extract/etc/fonts/*

$ sudo cp -r /tmp/extract/etc/fonts/* /etc/fonts/

Logout and log back in to see the change.

Using CSS to Make Your HTML Page Printer Friendly

Besides for styling for regular media (such as screen / pc monitor), css can also be used to make the web layout suitable for printing.
Perhaps you have a web page which has logo, menu, navigation, videos, and other things on it. Also there is a table or other information which you actually need. You would not want those irrelevant elements cluttering all over when you print it, right?

To solve it, simple use specific stylesheet for printing purpose and define it for "print" media. Here is an example of using external stylesheet for "print" media:


Next step is to eliminate those unnecessary elements using display: none. Here's an example:
div#menu {
display: none;
}

Now, as a bonus, I found out a way to make the default printing orientation as landscape. Here's it is:

@page {
size: landscape;
}

Another interesting thing that I found during my work is that !important rule defined in other css for any other media other than "print" will also effects the "print" media layout. I did not expect that, since I assume that !important rules defined for "screen" media, for example, would only effect that particular media. So, make sure you override the !important rule in the "print" css.

Django Blog Zinnia Limit User to Own Entry/Post

I just need a simple yet flexible django based blog application, and found django-blog-zinnia. I really admire how simple it is. The only major adjustment that I need is to limit user to his/her own blog entry/post.

Combining basic django (admin) knowledge with soome info from django-blog-zinnia documentation, I found a simple solution. Of course this is related with django-blog-zinnia's admin interface, and there is a documentation about how to make some modification to the EntryAdmin. Read more from here.

The idea is to create a django application and create admin.py on that app's top level directory (the same location with models.py).

Here is how my admin.py codes look like:
############# admin.py #################

from django.contrib import admin
from zinnia.models import Entry
from zinnia.admin.entry import EntryAdmin

class FilteredEntryAdmin(EntryAdmin):
    def queryset(self, request):
        qs = super(FilteredEntryAdmin, self).queryset(request)
        if request.user.is_superuser:
            return qs
        return qs.filter(authors=request.user)

admin.site.unregister(Entry)
admin.site.register(Entry, FilteredEntryAdmin)

############# end of admin.py #################

Remember to add the newly created app in settings.py INSTALLED_APPS, and make sure you put it below zinnia.

Django's documentation is quite complete and comprehensive. To get more information about ModelAdmin, queryset, etc, go to this link. From there I got the idea of overriding the queryset and use filter to limit the entry based on user.

(pip) Install Python Imaging Library (PIL)

I had a problem when I tried installing Python Imaging Library (PIL) version 1.1.7, within virtualenv, using pip. Actually, there was not any errors raised during installation, but when I used it in Django for the ImageField, I kept getting 'Upload a valid image. The file you uploaded was either not an image or a corrupted image' error message.
I figured out that the problem is due to the manual installation of PIL using pip, instead of using Debian/Ubuntu's apt-get.
So, the simplest solution would be using the
$ sudo apt-get install python-imaging

The thing is, I need PIL installed in my virtualenv environment. Getting some info from this link, I figured out a way to install PIL within virtualenv. In order to do that, some changes need to be done inside the PIL's setup.py script.
Step 0: sudo apt-get install libjpeg62 libjpeg62-dev
Step 1: get PIL source kit (here).
Step 2: untar, unzip the file
Step 3: locate setup.py from Step 2, and open that file for editing
Step 4: around line 200 (you should see other paths around there), add the following
add_directory(library_dirs, "/usr/lib/i386-linux-gnu")  # for 64 bit replace with /usr/lib/x86_64-linux-gnu
Step 5: tar and gzip the modified PIL source.
Step 6: activate your virtualenv, should you need it
Step 7: pip install

I made some test with my django-blog-zinnia project, by uploading some pictures and no more error messages.

Disable Network Manager and Use Script to Connect to WPA2 Wireless Connection

I would like some of the computers at my work to be able to connect to a specific wireless access point with WPA2 encryption and keep trying to connect (indefinitely) if the Access Point (AP) is not on yet. When the AP is up and running, those computers should be able to connect without any human interaction. There is a possibility that the AP is turned off for a while or restarted, due to some reason, and those computers should be able to reconnect automatically when the AP is back online again.

The default behavior on Ubuntu and Fedora is a little bit different (but far from my need). When a computer is already connected to AP, and the AP suddenly becomes unavailable for a while, there will be a WEP/WPA password dialog. User has to click the connect button again and if the AP is not available yet it will keep showing password dialog. When the user click cancel button, it will stop trying to connect to that particular AP.

Finally I figured out how to set the computer to keep trying to connect indefinitely and no matter how many times the AP becomes unavailable, it will keep doing it until it is (re)connected.

Here is how I did it:
The main issue is with the default behavior of the NetworkManager. I have not found any NetworkManager configuration so I could set it the way I explained earlier. Please let me know if someone knows how to do the exact thing that I need using NetworkManager.

The first step is to disable NetworkManager:
To disable it temporarily, use the following command:
$ sudo service network-manager stop

To permanently disable NetworkManager, do the following:
edit /etc/init/network-manager.conf
comment "start on...", so it would be like the following:
# start on

Next step is to create a wpa supplicant configuration file, do the following:
Create "/etc/wpa_supplicant/wpa_supplicant.conf" file and make it executable.
Put the following configurations inside it:
########### wpa_supplicant.conf ############


ctrl_interface=/var/run/wpa_supplicant

network={
    ssid="your_wlan_essid"
    scan_ssid=1
    key_mgmt=WPA-PSK
    psk="your_wpa2_passphrase"
}


########### end of wpa_supplicant.conf ############

Now, I am not really comfortable to put my plain text wpa2 passphrase inside the config file. So, I use the following method:
$ wpa_passphrase your_wlan_essid your_plain_text_passphrase

Use the output of the command above for your encrypted passphrase

Third step is to edit the /etc/network/interfaces configuration file.
Here is how my config file looks:
############# /etc/network/interfaces ###########

auto lo
iface lo inet loopback

auto wlan0
iface wlan0 inet dhcp
   wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
   wpa-action /etc/wpa_supplicant/action_wpa.sh

############# end of /etc/network/interfaces ###########

Notice that /etc/wpa_supplicant/wpa_supplicant.conf is the configuration file we just created previously, and /etc/wpa_supplicant/action_wpa.sh should exist from the default ubuntu (11.10) installation.

Now for the test:
Test #1
Make sure your AP is on.
Restart your computer, and after you logged in, you should be connected to AP.

Test #2

Make sure your AP is off.
Restart your computer, and after you logged in, you should not be connected to AP.

Turn on your AP.
Not long after that your computer should be connected to AP.

Test #2
Make sure your AP is on and your computer is connected to AP.
Turn off your AP.
Your computer should be disconnected from AP.
Wait for a while to be sure that your computer is not connected and the retry connection attempt failed.

Turn back your AP on.
Not long after that your computer should be reconnected to AP.


Now, this is exactly what I want for my setting at work.

Python SimpleHTTPServer

I needed to do some test and share a file on different machine. Fortunately there is a very simple solution for what I need, which is using Python's SimpleHTTPServer.
What I needed to do is simply go to the directory where the file is and then execute the following command:

$ python -m SimpleHTTPServer

Executing above command without arguments will run the webserver on 0.0.0.0:8000.

For more information just click here.

Disable Google Chrome Developer Tools

Finally, I found a way to disable google chrome developer tools. I need it because I have some browser based application that I need to deploy, and google chrome is really suitable for my need. The only thing that prevented me doing so is the dev tools. I don't want user to play around with the dev tools, do some hacking and stuff. No way!
Now, here is how to disable devtools:
Edit ~/.config/google-chrome/Default/Preferences
On Mac OSX: ~/Library/Application Support/Google/Chrome/Default
Add "disabled": true to the "devtools" key, so it becomes like the following:
"devtools": { "disabled": true, "split_location": 256 }
Save it, and restart browser. Done!
Here is where I originally found the solution.

Ubuntu 11.10 Virtual NIC IP Workaround

The normal way of adding virtual nic ip is by setting it in /etc/network/interfaces, like the following:

auto eth0:0
iface eth0:0 inet static
address 192.168.0.111
netmask 255.255.255.0
broadcast 192.168.0.255
gateway 192.168.0.1

Unfortunately, there is a known bug which prevent such configuration to work. Take a look here.

The first workaround that I figured out is by using /etc/rc.local script.

Add the following lines before exit(0) inside that script:

#################################

sleep 4  # This due to concurrency, set some delay in order for the networking to be set up properly first

ifconfig eth0:0 192.168.0.111 netmask 255.255.255.0 broadcast 192.168.0.255

# or use below command to replace above ifconfig command

# ip addr add 192.168.0.111 dev eth0

#################################

The second workaround is, IMHO, better. No sleep hack necessary, since upstart script is used instead of rc.local script.

Create /etc/init/network-interface-virtual.conf (or use different name as you like), with the following lines inside:

#################################

description "configure virtual network device"

respawn

console none

start on (net-device-up IFACE=eth0)

stop on runlevel [!12345]

script

exec ip addr add 192.168.0.111 dev eth0

# or use ifconfig command below

# ifconfig eth0:0 192.168.0.111 netmask 255.255.255.0 broadcast 192.168.0.255

end script

#################################

Reboot your system if necessary and everything is set and the virtual nic ip is ready to use. Oh, one more thing. There is also a bug (I think), which makes the virtual nic ip not showing with the ifconfig -a command. Just ping the virtual ip address to test whether it's up or not. 

Ubuntu 11.10 and Realtek (RTL8111/RTL8168) NIC driver problem

Finally, I figured out the problem that I had with network connection. The problem was with Realtek (RTL8111/RTL8168) NIC, which somehow detected differently by ubuntu 11.10 and uses r8169 kernel module driver.
To be sure, do check your NIC using:
$ sudo lscpi

You should see something like ...Realtek (RTL8111/RTL8168)...
Next step is to remove the r8169 module using:
$ rmmod r8169

If necessary install build-essential, etc:
$ sudo apt-get install build-essential linux-headers-$(uname -r) linux-source

Then, download the proper driver for that particular NIC from here. The file name should be something like r8168-8.027.00.tar.bz2.
Uncompress the file, and run the shell script file, as root:
# ./autorun.sh

Check the newly compiled driver:

$ sudo lsmod | grep r8168
$ ifconfig -a


Blacklist the r8169 module driver, by editing /etc/modprobe.d/blacklist.conf, add:
#blacklist r8169 driver
blacklist r8169

Reboot your system if necessary.

netcat as WebServer

I needed to have a (dummy) web server which gives only 201 CREATED response, for testing purpose. Instead of using real web server, I found a simpler way to achieve that. Using netcat like the following is enough for me to do some testing on my programs.

$ nc -C -l 8080 -q 1 << EOF
HTTP/1.0 201 CREATED
Content-Type: text/html
EOF

Interacting with Django Tastypie API Using Python Librarys

If you already have some django web-app, consider extending its functionality with django-tastypie. It has pretty decent docs as well. With django-tastypie you can create RESTful Web Service. I am not going to talk about django-tastypie in this post, instead I would like to share how to interact with django-tastypie api using python's urllib and urllib2 libraries.

The following codes gives some idea about how to get data using the API, with ApiKeyAuthentication method.

import urllib, urllib2

api_url = 'http://localhost:8080/api/app1/model1/?username=apiuser1&api_key=apikey_for_apiuser1'
http_req = urllib2.Request(api_url, headers={'ACCEPT': 'application/json, text/html'})
try:
    resp = urllib2.urlopen(http_req)
    contents = resp.read()
    print contents
except urllib2.HTTPError, error:
    print he.getcode()


The codes below gives some idea about how to create data using the API, with ApiKeyAuthentication method.


import urllib, urllib2
import simplejson as json


api_url = 'http://localhost:8080/api/app1/model1/?username=apiuser1&api_key=apikey_for_apiuser1'
json_post = json.dumps({"model_name": "TEST13"})
http_req = urllib2.Request(api_url, json_post, {'Content-Type': 'application/json', 'ACCEPT': 'application/json, text/html'})
try:
response = urllib2.urlopen(http_req)
print response.read()
print response.getcode()
print response.info()
print response.msg
except urllib2.HTTPError as he:
print he.getcode()
print he.info()
print he.msg
except urllib2.URLError as ue:
print ue

Both examples using json data format. Remember to set the appropriate Content-Type and ACCEPT header data if using different data format.
If it is necessary to use https secure connection and proxy, refer to this post to get more information.

Distributing Large Files - Implementation Part 1

This post is related to my previous post about distributing large files http://murzal-arsya.blogspot.com/2011/11/distributing-large-files.html.

Finally I implemented my distribution strategy as I explained in my previous post. Let me give a little bit info about the infrastructure.

1 Server:
Brand/Type: IBM System x3650 M3
OS: Centos 6
Service: Samba
NIC: 2 Gigabit

1 Switch: HP Procurve 1810 24g (Gigabit)

10 Clients:
Brand/Type: HP Pavilion HPE
OS: Windows 7
Apps: QGIS
NIC: 1 Gigabit


I have samba configured as shared storage. Within the share folder I put the large image files to be distributed to the clients. My strategy is creating a chain distribution, from the Server to PC-01, PC-01 to PC-02, and so on, until PC-10. I choose robocopy for its simplicity and it does the job well.
So, I created a batch file

Since I need to access the shared folder, I need to make sure that the connection could be established using correct user for authentication. In order to do that, before I run robocopy, I use net use command for authenticating.
>net use \\SERVER\share_images /USER:user1 password1
then I could execute robocopy with something like the following:
>robocopy /E /PURGE /COPY:DAT \\SERVER\share_images C:\images

The most ideal way to trigger the synchronization is when there is change in each share folder, and queuing the sync processes when there are multiple changes detected. It is beyond the scope of this post.

Python HTTPS POST Data Over Proxy

Posting HTTPS POST data over proxy is pretty simple in python. Everything is done mostly with urllib2. The only thing that is done outside urllib2 is when encoding the POST data, which uses urllib.urlencode.
Here is an example:


import urllib, urllib2

PROXY = urllib2.ProxyHandler({'https': 'http://your_proxy:your_proxy_port/'})
PROXY_OPENER = urllib2.build_opener(PROXY)
urllib2.install_opener(PROXY_OPENER)

URL = 'https://www.post-test-site.com/post.php'
POST = {'data1' : 'Your Data1', 'data2' : 'Your Data2'}

POST_DATA = urllib.urlencode(POST)
HTTPS_REQ = urllib2.Request(URL, POST_DATA)
RESP = urllib2.urlopen(HTTPS_REQ)
OUTPUT = RESP.read()

print OUTPUT

When your proxy needs authentication, then you need to use:
urllib2.HTTPBasicAuthHandler and/or urllib2.HTTPPasswordMgrWithDefaultRealm

Configure DHCPd to Work on Certain NIC(s)

Your server has more than one network interfaces and you need DHCP server to associated with only one of them.

Here is how you configure it on Ubuntu:

# File: /etc/default/dhcp3-server
INTERFACES="eth0"
or
INTERFACES="eth0 eth1"

Onn Redhat/Fedora/Centos:
# File: /etc/sysconfig/dhcpd
DHCPDARGS=eth1
or
DHCPDARGS="eth0 eth1"

Reference:
http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-dhcp-configuring-server.html

Distributing Large Files

One of the requirements of my new project is to be able to distribute large files to 10-15 clients as fast as possible. Fortunately, all (server and clients) NICs supports Gigabit connectivity.
I tested it first simply by copying large data from server to many clients simultaneously. Of course, it was a disaster. While theoretically the copying speed would decrease proportional to the number of clients, in reality the network connection between clients and server, and also between clients got disrupted. Network was down for sometimes. I did not make any further investigation on what was happening, all I know is this is far from the desired speed even if it was going well.
What's next? I gather some info's on how to distribute large files over LAN. Some suggest bittorrent, parallel filesystem, multicast copy, (win) mesh, and some other way. My concern is that I have to use a solution which is as simple as possible, since there would not be any expert IT guy around in the future.
While bittorrent sounds pretty promising, I had to leave out this option. Not only because of the capability of the human resources that will maintain the system, also it would take some times to prepare the large file to be distributed by bittorrent. Another thing is that small torrent packets could really saturate the network.

About parallel filesystem, I have to ditch this one too, because this would be more complicated and confusing for the basic admin guy. Also I need to mention that all clients are Windows 7, not really sure there are stable parallel filesystem support

Multicast copy should be interesting, but could not find any apps for Windows 7.

So, I figured that I need to make my own simple solution. The goal here is to distribute large files, around 1-5 GigaBytes each, without overloading the connection between source (server) and target (pc).

I simply came up with creating some kind of chain distribution. Transfer file from Server to PC1, and from PC1 to PC2, and so on. Doing this, I could use all of the available incoming gigabit bandwidth as well as the outgoing one. I tested it and worked quite well. Of course there will be a problem if one of the PC is down, then the distribution could not be continued to the rest of the PCs after that. So, rerouting must be done either manually or automatically.

As for the tool to copy the data, one can use simple copy, robocopy, synctoy, and other mechanism. I am planning to use command line robocopy.

I also plan to use load balancing technique using both NICs on the server. So, I can create 2 chains within the network, and perform the transfer simultaneously to those chains.

Permanently Set Network Interface Speed, Duplex or Auto Negotiate Settings on Linux

On RedHat (based) Linux:

Edit /etc/sysconfig/network-scripts/ifcfg-eth0 file for eth0 interface.
On my Fedora15, the file looks like the following:
/etc/sysconfig/network-scripts/ifcfg-Auto_wlan

So, open the corresponding file according to your need, then append the following line:
ETHTOOL_OPTS="speed 100 duplex full autoneg off"

Save the file, and restart network interface by executing the following command:
# /etc/init.d/network restart

The previous example is for setting the NIC to 100Mbps, full duplex and no auto negotiation.


If you want 1000Mbs use the following configuration:
ETHTOOL_OPTS="speed 1000 duplex full autoneg off"

or

ETHTOOL_OPTS="speed 1000 duplex full autoneg on"

For Ubuntu and perhaps Debian and other Debian based Linux, edit /etc/network/interfaces, and add pre-up statment, so it will look like something like the following:

auto eth0
iface eth0 inet static
pre-up /usr/sbin/ethtool -s eth0 speed 1000 duplex full

Refer to this link for more information.

For Ubuntu:
Edit /etc/network/interfaces and add the following line:
pre-up /usr/sbin/ethtool -s $IFACE autoneg off 100 duplex full

IBM System x3650 M3 Disk Configuration

I just finished working with IBM System x3650 M3. In the beginning, I had trouble with both Ubuntu Server 10.04.3 LTS and Centos 6 not detecting internal harddisks. Doing a little bit search, some informed that the problem is due to the driver, but that is not the case, since Ubuntu Server 10.04.3 LTS already have it.

Then I tried with Centos 6 and faced the same problem, no harddisk detected. Fortunately, I found some info in a forum mention about one need to configure the harddisk in the BIOS first. Aha. This is actually where the issue exist.

In order to configure the harddisks in the BIOS, just boot up the server and wait until a screen mention about Megaraid WebBIOS Configuration, press Ctrl-H.

Then, configure the harddisks as Array RAID. After that, just use Configuration Wizard to set them as as many Disk Group as needed. At this point, Centos 6 could detect the harddisk.

Find official guide from IBM, Installation and User's Guide - IBM ServeRAID-M Software (WebBIOS, MegaRAID Storage Manager, and MegaCLI), here.