build your own solr/hadoop bigdata cluster from old parts

I wanted to be able to compute some big data problems, such as reading and understanding a big subset of the internet, simulating a human brain, AI , machine learning, speech and natural language recognition, behaviour forecasting, data compression etc.

I chose Solr as a big data processing technology as it integrated and packaged lots of other interesting technologies and I hoped to use them and hadoop as needed.

I wanted to fully appreciate and test the sharding and replication features so I need a cluster of at least 2 machines.

I have an old HP laptop with 2Gbytes of RAM but limited disk space so I upgraded the disk space adding an external USB hard disk of 300Gbytes.
I run linux (Fedora 19) on this laptop and have it wired to my ethernet router on a local home network. It runs continuously in the attic.

I also have an old shuttle computer again with 2Gbytes of RAM and again limited disk space so I added another external USB hard disk of 300Gbytes.

These old usb external disks are very slow but they were surplus as I have moved my personal data to a 1TByte USB drive and so could reformat the old drives as ext4 filesystems dedicated to my big data.

The plugin USB drives automatically are mounted owned by the user who is logged in. This was not convenient for a continuous server so I created a mount point
mkdir /media/bigdata
chmod og+rwx /media/bigdata

and added the following line in /etc/fstab to mount them at boot as normal system drives.

/dev/sdb1   /media/bigdata  ext4  defaults   1   1

The old shuttle had lost its dvd and CDROM drive so I installed the fedora 20 using a bootable USB memory card drive. This was easy to make as you just download an iso image of fedora 20 then use dd to copy the iso image to the USB memory card drive. It overwrites everyhing including partitions so you specify the main device such as /dev/sdb and not a partition such as /dev/sdb1. Having looked at the instructions at http://docs.fedoraproject.org/en-US/Fedora/20/html/Installation_Guide/Making_USB_Media-UNIX_Linux.html#Making_USB_Media-UNIX_Linux-RHEL_Fedora-dd

I found that sdb was my usb memory drive so I did
dd if=fedora.iso of=/dev/sdb

After installing fedora 20 on the old shuttle, I found it convenient to boot in text mode rather than graphical as I did not have a monitor in the attic.
I followed
 geek.org/2013/11/03/how-to-with-systemd-fedora-19-text-boot-boot-to-console-mode/
to get the text boot. I could then login as root blind and shutdown if needed.

As it was a server, I wanted the network to be up at boot time and statically configured and so I chose a free IP address outside the range of my DHCP dynamic IP range (192.168.1.100-200) and found that for this particular hardware my ethernet was called enp2s0 and I edited the file
/etc/sysconfig/network-scripts/ifcfg-enp2s0 to contain:-

TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp2s0
UUID=21847aef-6a26-4add-a1ee-9882f159726b
ONBOOT=yes
HWADDR=00:30:1B:BA:37:E0
PEERDNS=yes
PEERROUTES=yes
IPADDR=192.168.1.73
BROADCAST=192.168.1.255
NETWORK=192.168.1.0
GATEWAY=192.168.1.254
DNS1=8.8.8.8
DNS2=8.8.4.4
DOMAIN=cycom.co.uk

The above lines in blue are changes or additions I made. The gateway was suitable for my BT HomeHub router.
My DNS was checked correct to use google public dns by looking at /etc/resolv.conf which had content:-
[root@localhost network-scripts]# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cycom.co.uk.
nameserver 8.8.8.8
nameserver 8.8.4.4

but make sure this doesn't get overwritten later. For background on my dns, see my previous blog
http://christopher-turner.blogspot.co.uk/2014/06/how-to-bypass-bt-unreliable-nameservers.html

The ZooKeeper component of solr cloud will use names (not ipaddresses) to identify the hosts in the cluster and for each machine to work out which one it is in a list of hosts so make sure that each host has unique name and that name resolution will work. On fedora 20 this is best achieved using entries in /etc/hosts and also changing /etc/hostname to a matching entry.

E.g. I have a 3 host cluster so on each host the /etc/hosts has this content:-

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.72 clusternode1
192.168.1.73 clusternode2
192.168.1.74 clusternode3

Installing JBOSS wildfly application server

Solr needs Java so first get java.

yum install java-1.7.0-openjdk

I also wanted a full J2EE application server and SOLR can run inside such a container so I chose JBOSS (some of my servers have "jboss as 7" but for this new build I tried jboss wildfly)

yum install wildfly

This drags in a huge selection of java technologies including solr3 but I dont think it will cause any conflict with the latest solr 4.9 which we load later. 

I want to run the solr as a cloud with one solr instance per machine and a zookeeper thread for each machine and I would rather not use an extra JVM for this (NO SUCCESS YET with wildfly) so I will enable the internal zookeeper server contained inside solr using the zkRun system property. You cannot set a system property using web.xml so set it in JBoss Wildfly.

cd /etc/wildfly/standalone

Edit the standalone.xml file to add the following system property snippet immediately following the extensions element. Note the zKRun value must be customised for each node:-

    </extensions> 
    <system-properties>  
        <property name="zkRun" value="clusternode3:9080"/> 
        <property name="zkHost" value="clusternode1:9080,clusternode2:9080,clusternode3:9080"/> 
    </system-properties> 

Set wildfly to start on system boot

systemctl enable wildfly.service

and lets start it now.

systemctl start wildfly.service

For me (wildfly-8.1.0-2.fc20.noarch) there was a bug with some file links in wildfly which prevented startup. /var/log/messages showed :-

java: Uncaught exception org.jboss.modules.ModuleLoadError in method javax.management.MBeanServerFactory.newMBeanServer()

and in /var/log/wildfly/standalone/server.log showed :-
java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main/jboss-marshalling-river.jar (No such file or directory)

To fix I had to do the following:-
cd /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main
rm -f jboss-marshalling-river.jar
ln -s /usr/share/java/jboss-marshalling/jboss-marshalling-river.jar jboss-marshalling-river.jar

try another start
systemctl start wildfly.service

This time it worked. We can see the wildfly listening on port 8080.
[root@localhost standalone]# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN    
tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:35495           0.0.0.0:*               LISTEN    
tcp6       0      0 :::111                  :::*                    LISTEN    
tcp6       0      0 :::34832                :::*                    LISTEN    
tcp6       0      0 :::22      

Installing Solr

Go to the place where you have plenty of disk space. For me this was the mounted 300Gbyte disk.

cd /media/bigdata

Download the latest Solr.

wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/lucene/solr/4.9.0/solr-4.9.0.tgz

and extract

tar xvzf solr-4.9.0.tgz

The solr docs will now be able to tell you how to run in solr embedded jetty http server but I want to use the wildfly so we need to edit the web.xml file that is contained in solr.war in order to set the properties for an absolute file path for solr home (as well as upgrade guava library).

First copy the solr.war to a tmp directory
mkdir ~/tmp
mkdir ~/tmp/solr
cd solr-4.9.0/example/webapps
cp solr.war ~/tmp/solr
cd ~/tmp/solr
unzip -o solr.war
cd WEB-INF/lib
wget http://search.maven.org/remotecontent?filepath=com/google/guava/guava/17.0/guava-17.0.jar -O guava-17.0.jar
rm -f guava-14.0.1.jar

Now edit the WEB-INF/web.xml  and uncomment the env-entry xml to modify it as follows:-

   <!-- People who want to hardcode their "Solr Home" directly into the
       WAR File can set the JNDI property here...
   -->
   <env-entry>
       <env-entry-name>solr/home</env-entry-name>
       <env-entry-value>/media/bigdata/solr-4.9.0/example/solr</env-entry-value>
       <env-entry-type>java.lang.String</env-entry-type>
    </env-entry>


Now replace the files in the war:-
cd ~/tmp/solr
rm -f solr.war
zip -r solr.war *

The wildfly server runs as user wildfly so we must make our solr files owned by wildfly.

cd /media
chown -R wildfly:wildfly bigdata


Now we are ready to deploy the solr.war to the wildfly deployment directory.

cd /usr/share/wildfly/standalone/deployments
cp ~/tmp/solr/solr.war .

Check the logs at /var/log/standalone/server.log

At this point I could see another wildfly error as :-
Caused by: org.jboss.modules.xml.XmlPullParserException: Failed to add resource root 'resteasy-json-p-provider-jandex.jar' at path 'resteasy-json-p-provider-jandex.jar' (position: END_TAG seen ...ar\"/>\\n\\t<resource-root path=\"resteasy-json-p-provider-jandex.jar\"/>... @29:61) caused by: java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main/resteasy-json-p-provider-jandex.jar (No such file or directory)"}}

So to fix this I did the following :-
cd  /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main
ln -s /usr/share/java/resteasy/resteasy-json-p-provider-jandex.jar resteasy-json-p-provider-jandex.jar

Then to trigger a redeploy:-

cd /usr/share/wildfly/standalone/deployments
rm -f solr.war.failed

If you see errors such as :-
Caused by: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
Then you need to check that you managed to upgrade guava library in the solr.war as detailed above.

 A successful solr deployment can be accessed at http://localhost:8080/solr

There are still unresolved security problems running in wildfly and the wildfly will not stop while running the embedded zookeeper so had to kill -9 processid.

After stopping all the wildfly processes, I used the following commands to successfully start all 3 nodes (one on each node).

On clusternode3:-
java -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -DzkRun=clusternode3:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar

On clusternode2:-
java -DzkRun=clusternode2:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar

On clusternode1:-
java -DzkRun=clusternode1:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar

how to configure secure openvpn tunnels to link my web server and my JBOSS/NetKernel/solr/hadoop cluster nodes on fedora 20

I have a virtual linux server in the cloud with my web server (nginx) running on it. I am using nginx to proxy certain web urls to Java based application servers (JBOSS, NetKernel, Apache SOLR, Hadoop etc) running on other nodes. I would like clients to use HTTPS SSL secured connections to the web server and to have the SSL certificates and config all terminating at the web server. The proxy connection onward to the Java cluster nodes and any connections between Java cluster nodes I would like to be secured with a VPN. This will mean I can relax the security and encryption settings of the Java nodes (e.g. only accept network traffic from the vpn or local network.)

First I install openvpn on each node and web server host machine.

yum install openvpn

It installed openvpn-2.3.2-4.fc20.x86_64 on my fedora 20 cloud web server machine and
openvpn-2.3.2-4.fc19.x86_64 on my fedora 19 cluster node.

Now I create a shared static secret 2048 bit key on one machine.

cd /etc/openvpn

mkdir keys

cd keys

openvpn --genkey --secret static.key

chmod 0400 static.key 

and then login to the other nodes and copy the same key file

cd /etc/openvpn

mkdir keys

cd keys

scp root@cloud.cycom.co.uk:/etc/openvpn/keys/static.key .

There are sample configurations at /usr/share/doc/openvpn but I have been copying from another blog at http://www.linux.com/learn/tutorials/743590-secure-remote-networking-with-openvpn-on-linux

On my server (www) which has a static public ip address 176.58.88.227, I create file /etc/openvpn/server.conf with content

# config for server
dev tun
ifconfig 10.0.0.1 10.0.0.2
secret /etc/openvpn/keys/static.key

and secure it with

chmod 0600 /etc/openvpn/server.conf

On the client java bigdata nodes, I create a file /etc/openvpn/bigdatanode.conf with content

# config for bigdatanode
dev tun
ifconfig 10.0.0.2 10.0.0.1
secret /etc/openvpn/keys/static.key
remote 176.58.88.227 

and secure it with

chmod 0600 /etc/openvpn/bigdatanode.conf

To test start openvpn in the foreground on both client and server specifying the relevant config file

openvpn /etc/openvpn/server.conf

openvpn /etc/openvpn/bigdatanode.conf

On the client we see a successful initialisation as follows:-

[root@localhost openvpn]# openvpn /etc/openvpn/bigdatanode.conf 
Fri Jul 11 16:30:09 2014 OpenVPN 2.3.2 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Sep 12 2013
Fri Jul 11 16:30:09 2014 TUN/TAP device tun0 opened
Fri Jul 11 16:30:09 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Fri Jul 11 16:30:09 2014 /usr/sbin/ip link set dev tun0 up mtu 1500
Fri Jul 11 16:30:10 2014 /usr/sbin/ip addr add dev tun0 local 10.0.0.2 peer 10.0.0.1
Fri Jul 11 16:30:10 2014 UDPv4 link local (bound): [undef]
Fri Jul 11 16:30:10 2014 UDPv4 link remote: [AF_INET]176.58.88.227:1194
Fri Jul 11 16:30:20 2014 Peer Connection Initiated with [AF_INET]176.58.88.227:1194
Fri Jul 11 16:30:21 2014 Initialization Sequence Completed
 

Now we need to make sure these processes start on boot automatically.
First control-C to kill each of the tests.

Then on the server machine:-

systemctl enable openvpn@server.service
systemctl start openvpn@server.service

And on the client bigdatanode machine :-

systemctl enable openvpn@bigdatanode.service
systemctl start openvpn@bigdatanode.service

You can now test a ping from either end e.g. from server,

[root@www system]# ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=60.1 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=59.3 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=59.1 ms

For more clients and servers you can create more tunnels by copying and renaming the server conf files
and separately enabling and starting them, however you must use additional ports for these tunnels, I suggest starting at 11194 (the default was 1194).
You must configure selinux to allow openvpn to bind to these non-standard ports as follows:-
semanage port -a -t openvpn_port_t -p udp 11194

The server2.conf file is now:-

# config for server
dev tun
ifconfig 10.0.0.3 10.0.0.4
port 11194
secret /etc/openvpn/keys/static.key

it would be enabled and started as :-

systemctl enable openvpn@server2.service
systemctl start openvpn@server2.service

and the extra client conf on another client would be:-

# config for bigdatanode

dev tun

ifconfig 10.0.0.4 10.0.0.3

secret /etc/openvpn/keys/static.key

remote '176.58.88.227 11194'

port 11194

The clients seemed not to create the tunnel until there was traffic originating at the client. The server on the other hand cannot create the tunnel as the packets will not reach client due to client firewall. So I had to provide firewall rules on clients and bthomehub to allow all udp packets to ports 1194 and 11194 routing them to the appropriate client fixed ip. Actually the server does not know where the clients are so that wont help. I must force some traffic from the clients to server (e.g. ping from client made it work).

The JBOSS  J2EE web application server is listening on port 8080 on all interfaces so we hope that includes the 10.0.0.2 interface. What I can do now is take off the local firewall rule which allows traffic on port 8080 from the public internet. I no longer need to open port 8080 as the VPN brings it through the firewall encapsulated and encrypted on an openvpn port.

On the server with the nginx proxy, I can use the new private 10.0.0.2 destination rather than the old public internet.

So a snippet from my nginx configuration is:-

   # another virtual host using mix of IP-, name-, and port-based configuration
    #
    server {
    #    listen       8000;
    #    listen       somename:8080;
        server_name  www.cyterm.com cyterm.com;
        root         /var/cycom/cyterm;
        location /solr/ {
                proxy_pass http://10.0.0.2:8080/solr/;
        }

        location / {
        }
    }

Having removed the 8080 port forward rule from BT HomeHub firewall and restarted nginx on the cloud server, I confirmed that solr can still be accessed via the public http://www.cyterm.com/solr/

How to bypass unreliable BT name servers on apple mac computer or windows computer

BT DNS (Domain name resolution service) has been unreliable and its BT Homehub does not permit the user setting of alternative DNS servers.

However each computer can be configured for alternative DNS servers individually and manually.

This is how you can choose to use Google's public DNS servers (thank you Google!).

For the apple mac :-

1. Select Network Preferences
2. Select Advanced.  
3. In the next window select DNS from the pull-down menu. 
4. Now select the + sign and a window with 0.0.0.0 comes up. Enter two ipv4 addresses consecutively: 8.8.8.8 and 8.8.4.4  then if you can enter two ipv6 addresses consecutively 2001:4860:4860::8888 and 2001:4860:4860::8844
5. Return to the first page and the DNS server IP addresses are shown.

See the apple mac screen shots below:-

For Microsoft Windows computers :-

1. Go the Control Panel.
2. Click Network and Internet, then Network and Sharing Center, and click Change adapter settings. See screenshot
   
3. Select the connection for which you want to configure Google Public DNS. For example:
   • To change the settings for an Ethernet connection, right-click Local Area Connection, and click Properties.
   • To change the settings for a wireless connection, right-click Wireless Network Connection, and click Properties
     
     .
   If you are prompted for an administrator password or confirmation, type the password or provide confirmation.
4. Highlight the ipv4 protocol item and click properties button
   
    
5. Select the radio button for manually supplied dns settings and enter the google ipv4 public dns values 8.8.8.8 and 8.8.4.4
   
    
6.  Highlight the ipv6 protocol item and click properties button
   
   .
7. Select the radio button for manually supplied dns settings and enter the google ipv6 public dns values 2001:4860:4860::8888 and 2001:4860:4860::8844
   
   
   Now restart the network connection if you wish to test the settings.
   There is similar information available at google at
   https://developers.google.com/speed/public-dns/docs/using
   
   For developers with the luxury of a linux server on their home network, they can alternatively set up their own DHCP server as outlined in my previous blog post