A recent windows or acer bios update caused my laptop to always boot windows and not my usual dual boot grub ubuntu linux menu. I found that booting while holding F2 brought up the bios menu and navigating to the boot section, I saw that Windows Boot Manager was selected as the first option in the boot sequence. I moved Window Boot Manager down below the HDD option using F5/F6 then exit bios saving changes. I found that it immediately restored my dual boot menu so I could choose my usual ubuntu.
Phew!
Sunday, 29 May 2016
Saturday, 19 December 2015
Built a website in S3 bucket with html form posts to another S3 bucket.
AWS is awesome! I just discovered that Amazon S3 buckets can be configured as static web sites so no web server required!
Of course static only refers to the web pages delivered. Those pages can still post form content to another S3 bucket also configured as a website.
The things that tripped me up were:-
a) Don't use the endpoint listed in the AWS console. Modify it to take out the region.
b) Specify an acl with the post. Otherwise even the bucket owner cannot download the content.
c) I have to keep setting read access for everyone every time I re-upload a file to the main site.
E.g. the posting form looks like:-
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form action="http://mysitenameuploads.s3.amazonaws.com/" method="post"
enctype="multipart/form-data">
Your name (any unique key): <input type="text" name="key" size="80" value="anonuser"/><br/>
Acl: <input type="text" name="acl" size="80" value="bucket-owner-full-control"/><br/>
Content: <input type="text" name="file" size="80" value="{ "body": "some json content"}"><br>
<input type="submit" value="Submit">
</form>
</body>
</html>
Of course static only refers to the web pages delivered. Those pages can still post form content to another S3 bucket also configured as a website.
The things that tripped me up were:-
a) Don't use the endpoint listed in the AWS console. Modify it to take out the region.
b) Specify an acl with the post. Otherwise even the bucket owner cannot download the content.
c) I have to keep setting read access for everyone every time I re-upload a file to the main site.
E.g. the posting form looks like:-
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form action="http://mysitenameuploads.s3.amazonaws.com/" method="post"
enctype="multipart/form-data">
Your name (any unique key): <input type="text" name="key" size="80" value="anonuser"/><br/>
Acl: <input type="text" name="acl" size="80" value="bucket-owner-full-control"/><br/>
Content: <input type="text" name="file" size="80" value="{ "body": "some json content"}"><br>
<input type="submit" value="Submit">
</form>
</body>
</html>
To read the posted data I am intending to turn on notifications for the uploads bucket and trigger AWS lamda code. Or I can download it later at leisure if there is no urgency in processing.
I have now implemented lambda functions triggered by s3 events (just empty with a log). The java one is terribly slow on first start and memory hungry.
REPORT RequestId: 97ad2da1-a835-11e5-9f10-271cbd9814df Duration: 2989.61 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 79 MB
The java one speeds up on subsequent requests to 2 ms (but still billed 100ms)
REPORT RequestId: cab829d5-a838-11e5-aa1b-c3dbfd807f5b Duration: 2.00 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 79 MB
So it looks as though I have a VM which gets swapped out when demand falls.
The python one is 1/2 the memory but still slower than I would like.
REPORT RequestId: f3b13d82-a82a-11e5-ba7f-c530ef1c7bd5 Duration: 197.21 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 43 MB
I have now implemented lambda functions triggered by s3 events (just empty with a log). The java one is terribly slow on first start and memory hungry.
REPORT RequestId: 97ad2da1-a835-11e5-9f10-271cbd9814df Duration: 2989.61 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 79 MB
The java one speeds up on subsequent requests to 2 ms (but still billed 100ms)
REPORT RequestId: cab829d5-a838-11e5-aa1b-c3dbfd807f5b Duration: 2.00 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 79 MB
So it looks as though I have a VM which gets swapped out when demand falls.
The python one is 1/2 the memory but still slower than I would like.
REPORT RequestId: f3b13d82-a82a-11e5-ba7f-c530ef1c7bd5 Duration: 197.21 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 43 MB
It also speeds up on repeated requests.
REPORT RequestId: 09325b69-a83b-11e5-82a2-51b3cc5c7850 Duration: 16.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 21 MB
The node version is the fastest and smallest. Worst case it has
REPORT RequestId: 17338845-a83e-11e5-9113-136c07ee4a8a Duration: 14.19 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 18 MB
After warmup it has
REPORT RequestId: 1eae0dc1-a83e-11e5-97a0-db389f3e9f5e Duration: 0.47 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 18 MB
So recommend node for implementation and try to do as much work per call as possible since you are billed 100ms anyway.
As Nginx and c++ can have very low memory footprint, maybe I should be using that and fastcgi.
REPORT RequestId: 09325b69-a83b-11e5-82a2-51b3cc5c7850 Duration: 16.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 21 MB
The node version is the fastest and smallest. Worst case it has
REPORT RequestId: 17338845-a83e-11e5-9113-136c07ee4a8a Duration: 14.19 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 18 MB
After warmup it has
REPORT RequestId: 1eae0dc1-a83e-11e5-97a0-db389f3e9f5e Duration: 0.47 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 18 MB
So recommend node for implementation and try to do as much work per call as possible since you are billed 100ms anyway.
As Nginx and c++ can have very low memory footprint, maybe I should be using that and fastcgi.
Tuesday, 30 December 2014
solved - dialog window too large for my laptop display on ubuntu
Some dialog windows are taller than 768 pixels and so you cannot see or press the OK button at the bottom. My acer aspire laptop running ubuntu 14.04 LTS has a screen resolution of 1366x768 pixels and no amount of ALT-mouse1 would persuade the window from adobe acrobat to move.
The solution was to use xrandr to create a virtual sized desktop and scale it to the physical display. The command which worked for me is:-
xrandr --fb 1920x1080 --output eDP1 --mode 1366x768 --panning 1920x1080 --scale 1.4055x1.4062
The solution was to use xrandr to create a virtual sized desktop and scale it to the physical display. The command which worked for me is:-
xrandr --fb 1920x1080 --output eDP1 --mode 1366x768 --panning 1920x1080 --scale 1.4055x1.4062
This worked immediately and the effect is that applications now think my display is 1920x1080 but it is scaled vertically to fit the screen (I never got mouse panning to work). The magic scale factor 1.4055 is 1920/1366 and the factor 1.4062 is 1080/768.
running xrandr with no arguments will show you the name and properties of your current display. E.g. mine now shows:-
cwturner@bigdataacer:~/bin$ xrandr
Screen 0: minimum 320 x 200, current 1920 x 1080, maximum 32767 x 32767
eDP1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 344mm x 194mm panning 1920x1080+0+0
1366x768 60.0*+
1360x768 59.8 60.0
1024x768 60.0
800x600 60.3 56.2
640x480 59.9
VGA1 disconnected (normal left inverted right x axis y axis)
HDMI1 disconnected (normal left inverted right x axis y axis)
DP1 disconnected (normal left inverted right x axis y axis)
HDMI2 disconnected (normal left inverted right x axis y axis)
VIRTUAL1 disconnected (normal left inverted right x axis y axis)
cwturner@bigdataacer:~/bin$
Friday, 11 July 2014
build your own solr/hadoop bigdata cluster from old parts
I wanted to be able to compute some big data problems, such as reading and understanding a big subset of the internet, simulating a human brain, AI , machine learning, speech and natural language recognition, behaviour forecasting, data compression etc.
I chose Solr as a big data processing technology as it integrated and packaged lots of other interesting technologies and I hoped to use them and hadoop as needed.
I wanted to fully appreciate and test the sharding and replication features so I need a cluster of at least 2 machines.
I have an old HP laptop with 2Gbytes of RAM but limited disk space so I upgraded the disk space adding an external USB hard disk of 300Gbytes.
I run linux (Fedora 19) on this laptop and have it wired to my ethernet router on a local home network. It runs continuously in the attic.
I also have an old shuttle computer again with 2Gbytes of RAM and again limited disk space so I added another external USB hard disk of 300Gbytes.
These old usb external disks are very slow but they were surplus as I have moved my personal data to a 1TByte USB drive and so could reformat the old drives as ext4 filesystems dedicated to my big data.
The plugin USB drives automatically are mounted owned by the user who is logged in. This was not convenient for a continuous server so I created a mount point
mkdir /media/bigdata
chmod og+rwx /media/bigdata
and added the following line in /etc/fstab to mount them at boot as normal system drives.
/dev/sdb1 /media/bigdata ext4 defaults 1 1
The old shuttle had lost its dvd and CDROM drive so I installed the fedora 20 using a bootable USB memory card drive. This was easy to make as you just download an iso image of fedora 20 then use dd to copy the iso image to the USB memory card drive. It overwrites everyhing including partitions so you specify the main device such as /dev/sdb and not a partition such as /dev/sdb1. Having looked at the instructions at http://docs.fedoraproject.org/en-US/Fedora/20/html/Installation_Guide/Making_USB_Media-UNIX_Linux.html#Making_USB_Media-UNIX_Linux-RHEL_Fedora-dd
I found that sdb was my usb memory drive so I did
dd if=fedora.iso of=/dev/sdb
After installing fedora 20 on the old shuttle, I found it convenient to boot in text mode rather than graphical as I did not have a monitor in the attic.
I followed
geek.org/2013/11/03/how-to-with-systemd-fedora-19-text-boot-boot-to-console-mode/
to get the text boot. I could then login as root blind and shutdown if needed.
As it was a server, I wanted the network to be up at boot time and statically configured and so I chose a free IP address outside the range of my DHCP dynamic IP range (192.168.1.100-200) and found that for this particular hardware my ethernet was called enp2s0 and I edited the file
/etc/sysconfig/network-scripts/ifcfg-enp2s0 to contain:-
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp2s0
UUID=21847aef-6a26-4add-a1ee-9882f159726b
ONBOOT=yes
HWADDR=00:30:1B:BA:37:E0
PEERDNS=yes
PEERROUTES=yes
IPADDR=192.168.1.73
BROADCAST=192.168.1.255
NETWORK=192.168.1.0
GATEWAY=192.168.1.254
DNS1=8.8.8.8
DNS2=8.8.4.4
DOMAIN=cycom.co.uk
The above lines in blue are changes or additions I made. The gateway was suitable for my BT HomeHub router.
My DNS was checked correct to use google public dns by looking at /etc/resolv.conf which had content:-
[root@localhost network-scripts]# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cycom.co.uk.
nameserver 8.8.8.8
nameserver 8.8.4.4
but make sure this doesn't get overwritten later. For background on my dns, see my previous blog
http://christopher-turner.blogspot.co.uk/2014/06/how-to-bypass-bt-unreliable-nameservers.html
The ZooKeeper component of solr cloud will use names (not ipaddresses) to identify the hosts in the cluster and for each machine to work out which one it is in a list of hosts so make sure that each host has unique name and that name resolution will work. On fedora 20 this is best achieved using entries in /etc/hosts and also changing /etc/hostname to a matching entry.
E.g. I have a 3 host cluster so on each host the /etc/hosts has this content:-
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.72 clusternode1
192.168.1.73 clusternode2
192.168.1.74 clusternode3
yum install java-1.7.0-openjdk
I also wanted a full J2EE application server and SOLR can run inside such a container so I chose JBOSS (some of my servers have "jboss as 7" but for this new build I tried jboss wildfly)
yum install wildfly
This drags in a huge selection of java technologies including solr3 but I dont think it will cause any conflict with the latest solr 4.9 which we load later.
I want to run the solr as a cloud with one solr instance per machine and a zookeeper thread for each machine and I would rather not use an extra JVM for this (NO SUCCESS YET with wildfly) so I will enable the internal zookeeper server contained inside solr using the zkRun system property. You cannot set a system property using web.xml so set it in JBoss Wildfly.
cd /etc/wildfly/standalone
Edit the standalone.xml file to add the following system property snippet immediately following the extensions element. Note the zKRun value must be customised for each node:-
</extensions>
<system-properties>
<property name="zkRun" value="clusternode3:9080"/>
<property name="zkHost" value="clusternode1:9080,clusternode2:9080,clusternode3:9080"/>
</system-properties>
Set wildfly to start on system boot
systemctl enable wildfly.service
and lets start it now.
systemctl start wildfly.service
For me (wildfly-8.1.0-2.fc20.noarch) there was a bug with some file links in wildfly which prevented startup. /var/log/messages showed :-
java: Uncaught exception org.jboss.modules.ModuleLoadError in method javax.management.MBeanServerFactory.newMBeanServer()
and in /var/log/wildfly/standalone/server.log showed :-
java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main/jboss-marshalling-river.jar (No such file or directory)
To fix I had to do the following:-
cd /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main
rm -f jboss-marshalling-river.jar
ln -s /usr/share/java/jboss-marshalling/jboss-marshalling-river.jar jboss-marshalling-river.jar
try another start
systemctl start wildfly.service
This time it worked. We can see the wildfly listening on port 8080.
[root@localhost standalone]# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9990 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:35495 0.0.0.0:* LISTEN
tcp6 0 0 :::111 :::* LISTEN
tcp6 0 0 :::34832 :::* LISTEN
tcp6 0 0 :::22
cd /media/bigdata
Download the latest Solr.
wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/lucene/solr/4.9.0/solr-4.9.0.tgz
and extract
tar xvzf solr-4.9.0.tgz
The solr docs will now be able to tell you how to run in solr embedded jetty http server but I want to use the wildfly so we need to edit the web.xml file that is contained in solr.war in order to set the properties for an absolute file path for solr home (as well as upgrade guava library).
First copy the solr.war to a tmp directory
mkdir ~/tmp
mkdir ~/tmp/solr
cd solr-4.9.0/example/webapps
cp solr.war ~/tmp/solr
cd ~/tmp/solr
unzip -o solr.war
cd WEB-INF/lib
wget http://search.maven.org/remotecontent?filepath=com/google/guava/guava/17.0/guava-17.0.jar -O guava-17.0.jar
rm -f guava-14.0.1.jar
Now edit the WEB-INF/web.xml and uncomment the env-entry xml to modify it as follows:-
<!-- People who want to hardcode their "Solr Home" directly into the
WAR File can set the JNDI property here...
-->
<env-entry>
<env-entry-name>solr/home</env-entry-name>
<env-entry-value>/media/bigdata/solr-4.9.0/example/solr</env-entry-value>
<env-entry-type>java.lang.String</env-entry-type>
</env-entry>
Now replace the files in the war:-
cd ~/tmp/solr
rm -f solr.war
zip -r solr.war *
The wildfly server runs as user wildfly so we must make our solr files owned by wildfly.
cd /media
chown -R wildfly:wildfly bigdata
Now we are ready to deploy the solr.war to the wildfly deployment directory.
cd /usr/share/wildfly/standalone/deployments
cp ~/tmp/solr/solr.war .
Check the logs at /var/log/standalone/server.log
At this point I could see another wildfly error as :-
Caused by: org.jboss.modules.xml.XmlPullParserException: Failed to add resource root 'resteasy-json-p-provider-jandex.jar' at path 'resteasy-json-p-provider-jandex.jar' (position: END_TAG seen ...ar\"/>\\n\\t<resource-root path=\"resteasy-json-p-provider-jandex.jar\"/>... @29:61) caused by: java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main/resteasy-json-p-provider-jandex.jar (No such file or directory)"}}
So to fix this I did the following :-
cd /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main
ln -s /usr/share/java/resteasy/resteasy-json-p-provider-jandex.jar resteasy-json-p-provider-jandex.jar
Then to trigger a redeploy:-
cd /usr/share/wildfly/standalone/deployments
rm -f solr.war.failed
If you see errors such as :-
Caused by: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
Then you need to check that you managed to upgrade guava library in the solr.war as detailed above.
A successful solr deployment can be accessed at http://localhost:8080/solr
There are still unresolved security problems running in wildfly and the wildfly will not stop while running the embedded zookeeper so had to kill -9 processid.
After stopping all the wildfly processes, I used the following commands to successfully start all 3 nodes (one on each node).
On clusternode3:-
java -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -DzkRun=clusternode3:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
On clusternode2:-
java -DzkRun=clusternode2:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
On clusternode1:-
java -DzkRun=clusternode1:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
I chose Solr as a big data processing technology as it integrated and packaged lots of other interesting technologies and I hoped to use them and hadoop as needed.
I wanted to fully appreciate and test the sharding and replication features so I need a cluster of at least 2 machines.
I have an old HP laptop with 2Gbytes of RAM but limited disk space so I upgraded the disk space adding an external USB hard disk of 300Gbytes.
I run linux (Fedora 19) on this laptop and have it wired to my ethernet router on a local home network. It runs continuously in the attic.
I also have an old shuttle computer again with 2Gbytes of RAM and again limited disk space so I added another external USB hard disk of 300Gbytes.
These old usb external disks are very slow but they were surplus as I have moved my personal data to a 1TByte USB drive and so could reformat the old drives as ext4 filesystems dedicated to my big data.
The plugin USB drives automatically are mounted owned by the user who is logged in. This was not convenient for a continuous server so I created a mount point
mkdir /media/bigdata
chmod og+rwx /media/bigdata
and added the following line in /etc/fstab to mount them at boot as normal system drives.
/dev/sdb1 /media/bigdata ext4 defaults 1 1
The old shuttle had lost its dvd and CDROM drive so I installed the fedora 20 using a bootable USB memory card drive. This was easy to make as you just download an iso image of fedora 20 then use dd to copy the iso image to the USB memory card drive. It overwrites everyhing including partitions so you specify the main device such as /dev/sdb and not a partition such as /dev/sdb1. Having looked at the instructions at http://docs.fedoraproject.org/en-US/Fedora/20/html/Installation_Guide/Making_USB_Media-UNIX_Linux.html#Making_USB_Media-UNIX_Linux-RHEL_Fedora-dd
I found that sdb was my usb memory drive so I did
dd if=fedora.iso of=/dev/sdb
After installing fedora 20 on the old shuttle, I found it convenient to boot in text mode rather than graphical as I did not have a monitor in the attic.
I followed
geek.org/2013/11/03/how-to-with-systemd-fedora-19-text-boot-boot-to-console-mode/
to get the text boot. I could then login as root blind and shutdown if needed.
As it was a server, I wanted the network to be up at boot time and statically configured and so I chose a free IP address outside the range of my DHCP dynamic IP range (192.168.1.100-200) and found that for this particular hardware my ethernet was called enp2s0 and I edited the file
/etc/sysconfig/network-scripts/ifcfg-enp2s0 to contain:-
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp2s0
UUID=21847aef-6a26-4add-a1ee-9882f159726b
ONBOOT=yes
HWADDR=00:30:1B:BA:37:E0
PEERDNS=yes
PEERROUTES=yes
IPADDR=192.168.1.73
BROADCAST=192.168.1.255
NETWORK=192.168.1.0
GATEWAY=192.168.1.254
DNS1=8.8.8.8
DNS2=8.8.4.4
DOMAIN=cycom.co.uk
The above lines in blue are changes or additions I made. The gateway was suitable for my BT HomeHub router.
My DNS was checked correct to use google public dns by looking at /etc/resolv.conf which had content:-
[root@localhost network-scripts]# cat /etc/resolv.conf
; generated by /usr/sbin/dhclient-script
search cycom.co.uk.
nameserver 8.8.8.8
nameserver 8.8.4.4
but make sure this doesn't get overwritten later. For background on my dns, see my previous blog
http://christopher-turner.blogspot.co.uk/2014/06/how-to-bypass-bt-unreliable-nameservers.html
The ZooKeeper component of solr cloud will use names (not ipaddresses) to identify the hosts in the cluster and for each machine to work out which one it is in a list of hosts so make sure that each host has unique name and that name resolution will work. On fedora 20 this is best achieved using entries in /etc/hosts and also changing /etc/hostname to a matching entry.
E.g. I have a 3 host cluster so on each host the /etc/hosts has this content:-
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.72 clusternode1
192.168.1.73 clusternode2
192.168.1.74 clusternode3
Installing JBOSS wildfly application server
Solr needs Java so first get java.yum install java-1.7.0-openjdk
I also wanted a full J2EE application server and SOLR can run inside such a container so I chose JBOSS (some of my servers have "jboss as 7" but for this new build I tried jboss wildfly)
yum install wildfly
This drags in a huge selection of java technologies including solr3 but I dont think it will cause any conflict with the latest solr 4.9 which we load later.
I want to run the solr as a cloud with one solr instance per machine and a zookeeper thread for each machine and I would rather not use an extra JVM for this (NO SUCCESS YET with wildfly) so I will enable the internal zookeeper server contained inside solr using the zkRun system property. You cannot set a system property using web.xml so set it in JBoss Wildfly.
cd /etc/wildfly/standalone
Edit the standalone.xml file to add the following system property snippet immediately following the extensions element. Note the zKRun value must be customised for each node:-
</extensions>
<system-properties>
<property name="zkRun" value="clusternode3:9080"/>
<property name="zkHost" value="clusternode1:9080,clusternode2:9080,clusternode3:9080"/>
</system-properties>
Set wildfly to start on system boot
systemctl enable wildfly.service
and lets start it now.
systemctl start wildfly.service
For me (wildfly-8.1.0-2.fc20.noarch) there was a bug with some file links in wildfly which prevented startup. /var/log/messages showed :-
java: Uncaught exception org.jboss.modules.ModuleLoadError in method javax.management.MBeanServerFactory.newMBeanServer()
and in /var/log/wildfly/standalone/server.log showed :-
java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main/jboss-marshalling-river.jar (No such file or directory)
To fix I had to do the following:-
cd /usr/share/wildfly/modules/system/layers/base/org/jboss/marshalling/river/main
rm -f jboss-marshalling-river.jar
ln -s /usr/share/java/jboss-marshalling/jboss-marshalling-river.jar jboss-marshalling-river.jar
try another start
systemctl start wildfly.service
This time it worked. We can see the wildfly listening on port 8080.
[root@localhost standalone]# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9990 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:35495 0.0.0.0:* LISTEN
tcp6 0 0 :::111 :::* LISTEN
tcp6 0 0 :::34832 :::* LISTEN
tcp6 0 0 :::22
Installing Solr
Go to the place where you have plenty of disk space. For me this was the mounted 300Gbyte disk.cd /media/bigdata
Download the latest Solr.
wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/lucene/solr/4.9.0/solr-4.9.0.tgz
and extract
tar xvzf solr-4.9.0.tgz
The solr docs will now be able to tell you how to run in solr embedded jetty http server but I want to use the wildfly so we need to edit the web.xml file that is contained in solr.war in order to set the properties for an absolute file path for solr home (as well as upgrade guava library).
First copy the solr.war to a tmp directory
mkdir ~/tmp
mkdir ~/tmp/solr
cd solr-4.9.0/example/webapps
cp solr.war ~/tmp/solr
cd ~/tmp/solr
unzip -o solr.war
cd WEB-INF/lib
wget http://search.maven.org/remotecontent?filepath=com/google/guava/guava/17.0/guava-17.0.jar -O guava-17.0.jar
rm -f guava-14.0.1.jar
Now edit the WEB-INF/web.xml and uncomment the env-entry xml to modify it as follows:-
<!-- People who want to hardcode their "Solr Home" directly into the
WAR File can set the JNDI property here...
-->
<env-entry>
<env-entry-name>solr/home</env-entry-name>
<env-entry-value>/media/bigdata/solr-4.9.0/example/solr</env-entry-value>
<env-entry-type>java.lang.String</env-entry-type>
</env-entry>
Now replace the files in the war:-
cd ~/tmp/solr
rm -f solr.war
zip -r solr.war *
The wildfly server runs as user wildfly so we must make our solr files owned by wildfly.
cd /media
chown -R wildfly:wildfly bigdata
Now we are ready to deploy the solr.war to the wildfly deployment directory.
cd /usr/share/wildfly/standalone/deployments
cp ~/tmp/solr/solr.war .
Check the logs at /var/log/standalone/server.log
At this point I could see another wildfly error as :-
Caused by: org.jboss.modules.xml.XmlPullParserException: Failed to add resource root 'resteasy-json-p-provider-jandex.jar' at path 'resteasy-json-p-provider-jandex.jar' (position: END_TAG seen ...ar\"/>\\n\\t<resource-root path=\"resteasy-json-p-provider-jandex.jar\"/>... @29:61) caused by: java.io.FileNotFoundException: /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main/resteasy-json-p-provider-jandex.jar (No such file or directory)"}}
So to fix this I did the following :-
cd /usr/share/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-json-p-provider/main
ln -s /usr/share/java/resteasy/resteasy-json-p-provider-jandex.jar resteasy-json-p-provider-jandex.jar
Then to trigger a redeploy:-
cd /usr/share/wildfly/standalone/deployments
rm -f solr.war.failed
If you see errors such as :-
Caused by: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
Then you need to check that you managed to upgrade guava library in the solr.war as detailed above.
A successful solr deployment can be accessed at http://localhost:8080/solr
There are still unresolved security problems running in wildfly and the wildfly will not stop while running the embedded zookeeper so had to kill -9 processid.
After stopping all the wildfly processes, I used the following commands to successfully start all 3 nodes (one on each node).
On clusternode3:-
java -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -DzkRun=clusternode3:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
On clusternode2:-
java -DzkRun=clusternode2:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
On clusternode1:-
java -DzkRun=clusternode1:9080 -DzkHost=clusternode1:9080,clusternode2:9080,clusternode3:9080 -DnumShards=2 -jar start.jar
how to configure secure openvpn tunnels to link my web server and my JBOSS/NetKernel/solr/hadoop cluster nodes on fedora 20
I have a virtual linux server in the cloud with my web server (nginx) running on it. I am using nginx to proxy certain web urls to Java based application servers (JBOSS, NetKernel, Apache SOLR, Hadoop etc) running on other nodes. I would like clients to use HTTPS SSL secured connections to the web server and to have the SSL certificates and config all terminating at the web server. The proxy connection onward to the Java cluster nodes and any connections between Java cluster nodes I would like to be secured with a VPN. This will mean I can relax the security and encryption settings of the Java nodes (e.g. only accept network traffic from the vpn or local network.)
First I install openvpn on each node and web server host machine.
yum install openvpn
It installed openvpn-2.3.2-4.fc20.x86_64 on my fedora 20 cloud web server machine and
openvpn-2.3.2-4.fc19.x86_64 on my fedora 19 cluster node.
Now I create a shared static secret 2048 bit key on one machine.
and then login to the other nodes and copy the same key file
First control-C to kill each of the tests.
Then on the server machine:-
systemctl enable openvpn@server.service
systemctl start openvpn@server.service
And on the client bigdatanode machine :-
systemctl enable openvpn@bigdatanode.service
systemctl start openvpn@bigdatanode.service
You can now test a ping from either end e.g. from server,
[root@www system]# ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=60.1 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=59.3 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=59.1 ms
For more clients and servers you can create more tunnels by copying and renaming the server conf files
and separately enabling and starting them, however you must use additional ports for these tunnels, I suggest starting at 11194 (the default was 1194).
You must configure selinux to allow openvpn to bind to these non-standard ports as follows:-
semanage port -a -t openvpn_port_t -p udp 11194
The server2.conf file is now:-
# config for server
dev tun
ifconfig 10.0.0.3 10.0.0.4
port 11194
secret /etc/openvpn/keys/static.key
The JBOSS J2EE web application server is listening on port 8080 on all interfaces so we hope that includes the 10.0.0.2 interface. What I can do now is take off the local firewall rule which allows traffic on port 8080 from the public internet. I no longer need to open port 8080 as the VPN brings it through the firewall encapsulated and encrypted on an openvpn port.
On the server with the nginx proxy, I can use the new private 10.0.0.2 destination rather than the old public internet.
So a snippet from my nginx configuration is:-
# another virtual host using mix of IP-, name-, and port-based configuration
#
server {
# listen 8000;
# listen somename:8080;
server_name www.cyterm.com cyterm.com;
root /var/cycom/cyterm;
location /solr/ {
proxy_pass http://10.0.0.2:8080/solr/;
}
location / {
}
}
Having removed the 8080 port forward rule from BT HomeHub firewall and restarted nginx on the cloud server, I confirmed that solr can still be accessed via the public http://www.cyterm.com/solr/
First I install openvpn on each node and web server host machine.
yum install openvpn
It installed openvpn-2.3.2-4.fc20.x86_64 on my fedora 20 cloud web server machine and
openvpn-2.3.2-4.fc19.x86_64 on my fedora 19 cluster node.
Now I create a shared static secret 2048 bit key on one machine.
cd /etc/openvpn
mkdir keys
cd keys
openvpn --genkey --secret static.key
chmod 0400 static.key
and then login to the other nodes and copy the same key file
cd /etc/openvpn
mkdir keys
cd keys
scp root@cloud.cycom.co.uk:/etc/openvpn/keys/static.key .
There are sample configurations at /usr/share/doc/openvpn but I have been copying from another blog at http://www.linux.com/learn/tutorials/743590-secure-remote-networking-with-openvpn-on-linuxOn my server (www) which has a static public ip address 176.58.88.227, I create file /etc/openvpn/server.conf with content
# config for server
dev tun
ifconfig 10.0.0.1 10.0.0.2
secret /etc/openvpn/keys/static.key
and secure it with
chmod 0600 /etc/openvpn/server.conf
On the client java bigdata nodes, I create a file /etc/openvpn/bigdatanode.conf with content
# config for bigdatanode
dev tun
ifconfig 10.0.0.2 10.0.0.1
secret /etc/openvpn/keys/static.key
remote 176.58.88.227
and secure it with
chmod 0600 /etc/openvpn/bigdatanode.conf
To test start openvpn in the foreground on both client and server specifying the relevant config file
openvpn /etc/openvpn/server.conf
openvpn /etc/openvpn/bigdatanode.conf
On the client we see a successful initialisation as follows:-
[root@localhost openvpn]# openvpn /etc/openvpn/bigdatanode.conf
Fri Jul 11 16:30:09 2014 OpenVPN 2.3.2 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Sep 12 2013
Fri Jul 11 16:30:09 2014 TUN/TAP device tun0 opened
Fri Jul 11 16:30:09 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Fri Jul 11 16:30:09 2014 /usr/sbin/ip link set dev tun0 up mtu 1500
Fri Jul 11 16:30:10 2014 /usr/sbin/ip addr add dev tun0 local 10.0.0.2 peer 10.0.0.1
Fri Jul 11 16:30:10 2014 UDPv4 link local (bound): [undef]
Fri Jul 11 16:30:10 2014 UDPv4 link remote: [AF_INET]176.58.88.227:1194
Fri Jul 11 16:30:20 2014 Peer Connection Initiated with [AF_INET]176.58.88.227:1194
Fri Jul 11 16:30:21 2014 Initialization Sequence Completed
Now we need to make sure these processes start on boot automatically.First control-C to kill each of the tests.
Then on the server machine:-
systemctl enable openvpn@server.service
systemctl start openvpn@server.service
And on the client bigdatanode machine :-
systemctl enable openvpn@bigdatanode.service
systemctl start openvpn@bigdatanode.service
You can now test a ping from either end e.g. from server,
[root@www system]# ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=60.1 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=59.3 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=59.1 ms
For more clients and servers you can create more tunnels by copying and renaming the server conf files
and separately enabling and starting them, however you must use additional ports for these tunnels, I suggest starting at 11194 (the default was 1194).
You must configure selinux to allow openvpn to bind to these non-standard ports as follows:-
semanage port -a -t openvpn_port_t -p udp 11194
The server2.conf file is now:-
# config for server
dev tun
ifconfig 10.0.0.3 10.0.0.4
port 11194
secret /etc/openvpn/keys/static.key
it would be enabled and started as :-
systemctl enable openvpn@server2.service
systemctl start openvpn@server2.service
systemctl start openvpn@server2.service
and the extra client conf on another client would be:-
# config for bigdatanode
dev tun
ifconfig 10.0.0.4 10.0.0.3
secret /etc/openvpn/keys/static.key
remote '176.58.88.227 11194'
port 11194
The clients seemed not to create the tunnel until there was traffic originating at the client. The server on the other hand cannot create the tunnel as the packets will not reach client due to client firewall. So I had to provide firewall rules on clients and bthomehub to allow all udp packets to ports 1194 and 11194 routing them to the appropriate client fixed ip. Actually the server does not know where the clients are so that wont help. I must force some traffic from the clients to server (e.g. ping from client made it work).
On the server with the nginx proxy, I can use the new private 10.0.0.2 destination rather than the old public internet.
So a snippet from my nginx configuration is:-
# another virtual host using mix of IP-, name-, and port-based configuration
#
server {
# listen 8000;
# listen somename:8080;
server_name www.cyterm.com cyterm.com;
root /var/cycom/cyterm;
location /solr/ {
proxy_pass http://10.0.0.2:8080/solr/;
}
location / {
}
}
Having removed the 8080 port forward rule from BT HomeHub firewall and restarted nginx on the cloud server, I confirmed that solr can still be accessed via the public http://www.cyterm.com/solr/
Monday, 7 July 2014
How to bypass unreliable BT name servers on apple mac computer or windows computer
BT DNS (Domain name resolution service) has been unreliable and its BT Homehub does not permit the user setting of alternative DNS servers.
However each computer can be configured for alternative DNS servers individually and manually.
This is how you can choose to use Google's public DNS servers (thank you Google!).
For the apple mac :-
For Microsoft Windows computers :-
However each computer can be configured for alternative DNS servers individually and manually.
This is how you can choose to use Google's public DNS servers (thank you Google!).
For the apple mac :-
- Select Network Preferences
- Select Advanced.
- In the next window select DNS from the pull-down menu.
- Now select the + sign and a window with 0.0.0.0 comes up. Enter two ipv4 addresses consecutively: 8.8.8.8 and 8.8.4.4 then if you can enter two ipv6 addresses consecutively 2001:4860:4860::8888 and 2001:4860:4860::8844
- Return to the first page and the DNS server IP addresses are shown.
For Microsoft Windows computers :-
- Go the Control Panel.
- Click Network and Internet, then Network
and Sharing Center, and click Change adapter settings. See screenshot
- Select the connection for which you want to configure
Google Public DNS. For example:
- To change the settings for an Ethernet connection, right-click Local Area Connection, and click Properties.
- To change the settings for a wireless connection,
right-click Wireless Network Connection, and click Properties .
- Highlight the ipv4 protocol item and click properties button
- Select the radio button for manually supplied dns settings and enter the google ipv4 public dns values 8.8.8.8 and 8.8.4.4
- Highlight the ipv6 protocol item and click properties button.
- Select the radio button for manually supplied dns settings and enter the google ipv6 public dns values 2001:4860:4860::8888 and 2001:4860:4860::8844
Now restart the network connection if you wish to test the settings.
There is similar information available at google at
https://developers.google.com/speed/public-dns/docs/using
For developers with the luxury of a linux server on their home network, they can alternatively set up their own DHCP server as outlined in my previous blog post
Saturday, 28 June 2014
How to bypass BT unreliable nameservers with your own fedora linux dhcp server
I was happily browsing via my BT Homehub broadband connection when for some reason sourceforge.net failed to be resolved in DNS.
I checked with
dig sourceforge.net
and sure enough no answer from nameserver.
The default nameserver was 192.168.1.254 which is my BT Homehub router.
Assuming this then delegated to BT nameservers dynamically supplied when the broadband connection was made, I looked up these nameservers from the homehub status pages (http://192.168.1.254) and tried them directly and still no answer.
However google nameservers were able to provide an answer.
dig @8.8.8.8 sourceforge.net
I tried rebooting the router and got different nameservers but still not fully functional for all domains.
Google was still working for most sites (but not for some bt community forums which google was showing might have a solution for me).
Sadly the nameservers are not configurable in the BT HomeHub.
I could configure my PC to not use the automatic DHCP supplied DNS servers for this session by editing /etc/resolv.conf to use google nameservers:-
# Generated by NetworkManager
domain cycom.co.uk
search cycom.co.uk cycom.co.uk.
nameserver 8.8.8.8
nameserver 8.8.4.4
(note microsoft users can do something similar by editing the IPV4 and IPV6 properties of their network connection as explained in my next blog post).
but /etc/resolv.conf was going to be overwritten by network manager next time I connect and I really wanted to continue to use DHCP for everything including from my android phones so with the temporary fix in place I started to configure a dhcp server on my local fedora linux server.
You do need a local linux server configured with a fixed (local network) ip on your home network (e.g. 192.168.1.72).
On this local server, I installed dhcpd with:-
yum install dhcp
Then I edit the config file /etc/dhcp/dhcp.conf to have this content:-
# dhcpd.conf
#
# Sample configuration file for ISC dhcpd
#
# option definitions common to all supported networks...
option domain-name "cycom.co.uk";
option domain-name-servers 8.8.8.8, 8.8.4.4;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;
option routers 192.168.1.254;
option domain-search "cycom.co.uk";
default-lease-time 6000;
max-lease-time 72000;
# Use this to enable / disable dynamic dns updates globally.
#ddns-update-style none;
# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;
# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.1.200;
}
and then enable and start the dhcpd service:-
systemctl enable dhcpd
systemctl start dhcpd
On the BT HomeHub router I disabled dhcp server leaving me with just the dhcp server on my linux server.
After testing with my android devices, I rebooted my linux laptop (this is the one where I had temporarily overridden the nameservers) and confirmed that the DHCP supplied values now matched google nameservers.
I can sleep much happier now knowing that my DNS is independent of the unreliable BT nameservice.
I checked with
dig sourceforge.net
and sure enough no answer from nameserver.
The default nameserver was 192.168.1.254 which is my BT Homehub router.
Assuming this then delegated to BT nameservers dynamically supplied when the broadband connection was made, I looked up these nameservers from the homehub status pages (http://192.168.1.254) and tried them directly and still no answer.
However google nameservers were able to provide an answer.
dig @8.8.8.8 sourceforge.net
I tried rebooting the router and got different nameservers but still not fully functional for all domains.
Google was still working for most sites (but not for some bt community forums which google was showing might have a solution for me).
Sadly the nameservers are not configurable in the BT HomeHub.
I could configure my PC to not use the automatic DHCP supplied DNS servers for this session by editing /etc/resolv.conf to use google nameservers:-
# Generated by NetworkManager
domain cycom.co.uk
search cycom.co.uk cycom.co.uk.
nameserver 8.8.8.8
nameserver 8.8.4.4
(note microsoft users can do something similar by editing the IPV4 and IPV6 properties of their network connection as explained in my next blog post).
but /etc/resolv.conf was going to be overwritten by network manager next time I connect and I really wanted to continue to use DHCP for everything including from my android phones so with the temporary fix in place I started to configure a dhcp server on my local fedora linux server.
You do need a local linux server configured with a fixed (local network) ip on your home network (e.g. 192.168.1.72).
On this local server, I installed dhcpd with:-
yum install dhcp
Then I edit the config file /etc/dhcp/dhcp.conf to have this content:-
# dhcpd.conf
#
# Sample configuration file for ISC dhcpd
#
# option definitions common to all supported networks...
option domain-name "cycom.co.uk";
option domain-name-servers 8.8.8.8, 8.8.4.4;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;
option routers 192.168.1.254;
option domain-search "cycom.co.uk";
default-lease-time 6000;
max-lease-time 72000;
# Use this to enable / disable dynamic dns updates globally.
#ddns-update-style none;
# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;
# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.1.200;
}
and then enable and start the dhcpd service:-
systemctl enable dhcpd
systemctl start dhcpd
On the BT HomeHub router I disabled dhcp server leaving me with just the dhcp server on my linux server.
After testing with my android devices, I rebooted my linux laptop (this is the one where I had temporarily overridden the nameservers) and confirmed that the DHCP supplied values now matched google nameservers.
I can sleep much happier now knowing that my DNS is independent of the unreliable BT nameservice.
Subscribe to:
Posts (Atom)