I chose Solr as a big data processing technology as it integrated and packaged lots of other interesting technologies and I hoped to use them and hadoop as needed.
I wanted to fully appreciate and test the sharding and replication features so I need a cluster of at least 2 machines.
I have an old HP laptop with 2Gbytes of RAM but limited disk space so I upgraded the disk space adding an external USB hard disk of 300Gbytes.
I run linux (Fedora 19) on this laptop and have it wired to my ethernet router on a local home network. It runs continuously in the attic.
I also have an old shuttle computer again with 2Gbytes of RAM and again limited disk space so I added another external USB hard disk of 300Gbytes.
These old usb external disks are very slow but they were surplus as I have moved my personal data to a 1TByte USB drive and so could reformat the old drives as ext4 filesystems dedicated to my big data.
The plugin USB drives automatically are mounted owned by the user who is logged in. This was not convenient for a continuous server so I created a mount point
mkdir /media/bigdata
chmod og+rwx /media/bigdata
and added the following line in /etc/fstab to mount them at boot as normal system drives.
/dev/sdb1 /media/bigdata ext4 defaults 1 1
The old shuttle had lost its dvd and CDROM drive so I installed the fedora 20 using a bootable USB memory card drive. This was easy to make as you just download an iso image of fedora 20 then use dd to copy the iso image to the USB memory card drive. It overwrites everyhing including partitions so you specify the main device such as /dev/sdb and not a partition such as /dev/sdb1. Having looked at the instructions at http://docs.fedoraproject.org/en-US/Fedora/20/html/Installation_Guide/Making_USB_Media-UNIX_Linux.html#Making_USB_Media-UNIX_Linux-RHEL_Fedora-dd
I found that sdb was my usb memory drive so I did
dd if=fedora.iso of=/dev/sdb
After installing fedora 20 on the old shuttle, I found it convenient to boot in text mode rather than graphical as I did not have a monitor in the attic.
I followed
geek.org/2013/11/03/how-to-with-systemd-fedora-19-text-boot-boot-to-console-mode/
to get the text boot. I could then login as root blind and shutdown if needed.
As it was a server, I wanted the network to be up at boot time and statically configured and so I chose a free IP address outside the range of my DHCP dynamic IP range (192.168.1.100-200) and found that for this particular hardware my ethernet was called enp2s0 and I edited the file
/etc/sysconfig/network-scripts/ifcfg-enp2s0 to contain:-
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp2s0
UUID=21847aef-6a26-4add-a1ee-9882f159726b
ONBOOT=yes
HWADDR=00:30:1B:BA:37:E0
PEERDNS=yes
PEERROUTES=yes
IPADDR=192.168.1.73
BROADCAST=192.168.1.255
NETWORK=192.168.1.0
GATEWAY=192.168.1.254
DNS1=8.8.8.8
DNS2=8.8.4.4
DOMAIN=cycom.co.uk
The above lines in blue are changes or additions I made. The gateway was suitable for my BT HomeHub router.
My DNS was checked correct to use google public dns by looking at /etc/resolv.conf which had content:-
[root@localhost network-scripts]# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cycom.co.uk.
nameserver 8.8.8.8
nameserver 8.8.4.4
but make sure this doesn't get overwritten later. For background on my dns, see my previous blog
http://christopher-turner.blogspot.co.uk/2014/06/how-to-bypass-bt-unreliable-nameservers.html
The ZooKeeper component of solr cloud will use names (not ipaddresses) to identify the hosts in the cluster and for each machine to work out which one it is in a list of hosts so make sure that each host has unique name and that name resolution will work. On fedora 20 this is best achieved using entries in /etc/hosts and also changing /etc/hostname to a matching entry.
E.g. I have a 3 host cluster so on each host the /etc/hosts has this content:-
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.72 clusternode1
192.168.1.73 clusternode2
192.168.1.74 clusternode3
Installing JBOSS wildfly application server
Solr needs Java so first get java.yum install java-1.7.0-openjdk
I also wanted a full J2EE application server and SOLR can run inside such a container so I chose JBOSS (some of my servers have "jboss as 7" but for this new build I tried jboss wildfly)
yum install wildfly
This drags in a huge selection of java technologies including solr3 but I dont think it will cause any conflict with the latest solr 4.9 which we load later.
I want to run the solr as a cloud with one solr instance per machine and a zookeeper thread for each machine and I would rather not use an extra JVM for this (NO SUCCESS YET with wildfly) so I will enable the internal zookeeper server contained inside solr using the zkRun system property. You cannot set a system property using web.xml so set it in JBoss Wildfly.
cd /etc/wildfly/standalone
Edit the standalone.xml file to add the following system property snippet immediately following the extensions element. Note the zKRun value must be customised for each node:-
</extensions>
<system-properties>
<property name="zkRun" value="clusternode3:9080"/>
<property name="zkHost" value="clusternode1:9080,clusternode2:9080,clusternode3:9080"/>
</system-properties>
Set wildfly to start on system boot
systemctl enable wildfly.service
and lets start it now.
systemctl start wildfly.service
For me (wildfly-8.1.0-2.fc20.noarch) there was a bug with some file links in wildfly which prevented startup. /var/log/messages showed :-
java: Uncaught exception org.jboss.modules.ModuleLoadError in method javax.management.MBeanServerFactory.newMBeanServer()
and in /var/log/wildfly/standalone/server.log showed :-
java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main/jboss-marshalling-river.jar (No such file or directory)
To fix I had to do the following:-
cd /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main
rm -f jboss-marshalling-river.jar
ln -s /usr/share/java/jboss-marshalling/jboss-marshalling-river.jar jboss-marshalling-river.jar
try another start
systemctl start wildfly.service
This time it worked. We can see the wildfly listening on port 8080.
[root@localhost standalone]# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9990 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:35495 0.0.0.0:* LISTEN
tcp6 0 0 :::111 :::* LISTEN
tcp6 0 0 :::34832 :::* LISTEN
tcp6 0 0 :::22
Installing Solr
Go to the place where you have plenty of disk space. For me this was the mounted 300Gbyte disk.cd /media/bigdata
Download the latest Solr.
wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/lucene/solr/4.9.0/solr-4.9.0.tgz
and extract
tar xvzf solr-4.9.0.tgz
The solr docs will now be able to tell you how to run in solr embedded jetty http server but I want to use the wildfly so we need to edit the web.xml file that is contained in solr.war in order to set the properties for an absolute file path for solr home (as well as upgrade guava library).
First copy the solr.war to a tmp directory
mkdir ~/tmp
mkdir ~/tmp/solr
cd solr-4.9.0/example/webapps
cp solr.war ~/tmp/solr
cd ~/tmp/solr
unzip -o solr.war
cd WEB-INF/lib
wget http://search.maven.org/remotecontent?filepath=com/google/guava/guava/17.0/guava-17.0.jar -O guava-17.0.jar
rm -f guava-14.0.1.jar
Now edit the WEB-INF/web.xml and uncomment the env-entry xml to modify it as follows:-
<!-- People who want to hardcode their "Solr Home" directly into the
WAR File can set the JNDI property here...
-->
<env-entry>
<env-entry-name>solr/home</env-entry-name>
<env-entry-value>/media/bigdata/solr-4.9.0/example/solr</env-entry-value>
<env-entry-type>java.lang.String</env-entry-type>
</env-entry>
Now replace the files in the war:-
cd ~/tmp/solr
rm -f solr.war
zip -r solr.war *
The wildfly server runs as user wildfly so we must make our solr files owned by wildfly.
cd /media
chown -R wildfly:wildfly bigdata
Now we are ready to deploy the solr.war to the wildfly deployment directory.
cd /usr/share/wildfly/standalone/deployments
cp ~/tmp/solr/solr.war .
Check the logs at /var/log/standalone/server.log
At this point I could see another wildfly error as :-
Caused by: org.jboss.modules.xml.XmlPullParserException: Failed to add resource root 'resteasy-json-p-provider-jandex.jar' at path 'resteasy-json-p-provider-jandex.jar' (position: END_TAG seen ...ar\"/>\\n\\t<resource-root path=\"resteasy-json-p-provider-jandex.jar\"/>... @29:61) caused by: java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main/resteasy-json-p-provider-jandex.jar (No such file or directory)"}}
So to fix this I did the following :-
cd /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main
ln -s /usr/share/java/resteasy/resteasy-json-p-provider-jandex.jar resteasy-json-p-provider-jandex.jar
Then to trigger a redeploy:-
cd /usr/share/wildfly/standalone/deployments
rm -f solr.war.failed
If you see errors such as :-
Caused by: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
Then you need to check that you managed to upgrade guava library in the solr.war as detailed above.
A successful solr deployment can be accessed at http://localhost:8080/solr
There are still unresolved security problems running in wildfly and the wildfly will not stop while running the embedded zookeeper so had to kill -9 processid.
After stopping all the wildfly processes, I used the following commands to successfully start all 3 nodes (one on each node).
On clusternode3:-
java -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -DzkRun=clusternode3:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
On clusternode2:-
java -DzkRun=clusternode2:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
On clusternode1:-
java -DzkRun=clusternode1:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
Thanks for your tip about resteasy-json-p-provider-jandex.jar
ReplyDelete