how to configure secure openvpn tunnels to link my web server and my JBOSS/NetKernel/solr/hadoop cluster nodes on fedora 20

I have a virtual linux server in the cloud with my web server (nginx) running on it. I am using nginx to proxy certain web urls to Java based application servers (JBOSS, NetKernel, Apache SOLR, Hadoop etc) running on other nodes. I would like clients to use HTTPS SSL secured connections to the web server and to have the SSL certificates and config all terminating at the web server. The proxy connection onward to the Java cluster nodes and any connections between Java cluster nodes I would like to be secured with a VPN. This will mean I can relax the security and encryption settings of the Java nodes (e.g. only accept network traffic from the vpn or local network.)

First I install openvpn on each node and web server host machine.

yum install openvpn

It installed openvpn-2.3.2-4.fc20.x86_64 on my fedora 20 cloud web server machine and
openvpn-2.3.2-4.fc19.x86_64 on my fedora 19 cluster node.

Now I create a shared static secret 2048 bit key on one machine.

cd /etc/openvpn

mkdir keys

cd keys

openvpn --genkey --secret static.key

chmod 0400 static.key 

and then login to the other nodes and copy the same key file

cd /etc/openvpn

mkdir keys

cd keys

scp root@cloud.cycom.co.uk:/etc/openvpn/keys/static.key .

There are sample configurations at /usr/share/doc/openvpn but I have been copying from another blog at http://www.linux.com/learn/tutorials/743590-secure-remote-networking-with-openvpn-on-linux

On my server (www) which has a static public ip address 176.58.88.227, I create file /etc/openvpn/server.conf with content

# config for server
dev tun
ifconfig 10.0.0.1 10.0.0.2
secret /etc/openvpn/keys/static.key

and secure it with

chmod 0600 /etc/openvpn/server.conf

On the client java bigdata nodes, I create a file /etc/openvpn/bigdatanode.conf with content

# config for bigdatanode
dev tun
ifconfig 10.0.0.2 10.0.0.1
secret /etc/openvpn/keys/static.key
remote 176.58.88.227 

and secure it with

chmod 0600 /etc/openvpn/bigdatanode.conf

To test start openvpn in the foreground on both client and server specifying the relevant config file

openvpn /etc/openvpn/server.conf

openvpn /etc/openvpn/bigdatanode.conf

On the client we see a successful initialisation as follows:-

[root@localhost openvpn]# openvpn /etc/openvpn/bigdatanode.conf 
Fri Jul 11 16:30:09 2014 OpenVPN 2.3.2 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Sep 12 2013
Fri Jul 11 16:30:09 2014 TUN/TAP device tun0 opened
Fri Jul 11 16:30:09 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Fri Jul 11 16:30:09 2014 /usr/sbin/ip link set dev tun0 up mtu 1500
Fri Jul 11 16:30:10 2014 /usr/sbin/ip addr add dev tun0 local 10.0.0.2 peer 10.0.0.1
Fri Jul 11 16:30:10 2014 UDPv4 link local (bound): [undef]
Fri Jul 11 16:30:10 2014 UDPv4 link remote: [AF_INET]176.58.88.227:1194
Fri Jul 11 16:30:20 2014 Peer Connection Initiated with [AF_INET]176.58.88.227:1194
Fri Jul 11 16:30:21 2014 Initialization Sequence Completed
 

Now we need to make sure these processes start on boot automatically.
First control-C to kill each of the tests.

Then on the server machine:-

systemctl enable openvpn@server.service
systemctl start openvpn@server.service

And on the client bigdatanode machine :-

systemctl enable openvpn@bigdatanode.service
systemctl start openvpn@bigdatanode.service

You can now test a ping from either end e.g. from server,

[root@www system]# ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=60.1 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=59.3 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=59.1 ms

For more clients and servers you can create more tunnels by copying and renaming the server conf files
and separately enabling and starting them, however you must use additional ports for these tunnels, I suggest starting at 11194 (the default was 1194).
You must configure selinux to allow openvpn to bind to these non-standard ports as follows:-
semanage port -a -t openvpn_port_t -p udp 11194

The server2.conf file is now:-

# config for server
dev tun
ifconfig 10.0.0.3 10.0.0.4
port 11194
secret /etc/openvpn/keys/static.key

it would be enabled and started as :-

systemctl enable openvpn@server2.service
systemctl start openvpn@server2.service

and the extra client conf on another client would be:-

# config for bigdatanode

dev tun

ifconfig 10.0.0.4 10.0.0.3

secret /etc/openvpn/keys/static.key

remote '176.58.88.227 11194'

port 11194

The clients seemed not to create the tunnel until there was traffic originating at the client. The server on the other hand cannot create the tunnel as the packets will not reach client due to client firewall. So I had to provide firewall rules on clients and bthomehub to allow all udp packets to ports 1194 and 11194 routing them to the appropriate client fixed ip. Actually the server does not know where the clients are so that wont help. I must force some traffic from the clients to server (e.g. ping from client made it work).

The JBOSS  J2EE web application server is listening on port 8080 on all interfaces so we hope that includes the 10.0.0.2 interface. What I can do now is take off the local firewall rule which allows traffic on port 8080 from the public internet. I no longer need to open port 8080 as the VPN brings it through the firewall encapsulated and encrypted on an openvpn port.

On the server with the nginx proxy, I can use the new private 10.0.0.2 destination rather than the old public internet.

So a snippet from my nginx configuration is:-

   # another virtual host using mix of IP-, name-, and port-based configuration
    #
    server {
    #    listen       8000;
    #    listen       somename:8080;
        server_name  www.cyterm.com cyterm.com;
        root         /var/cycom/cyterm;
        location /solr/ {
                proxy_pass http://10.0.0.2:8080/solr/;
        }

        location / {
        }
    }

Having removed the 8080 port forward rule from BT HomeHub firewall and restarted nginx on the cloud server, I confirmed that solr can still be accessed via the public http://www.cyterm.com/solr/