20 Nov 2015

Reverse Port Forwarding and how !

Recently had to work on a PoC that required spinning off Docker instances with custom configurations. The problem wasn't as much with the Docker aspect, which is probably up for another story sometime, as much as the fact that as luck had it, we couldn't commission a separate machine for this PoC, and I eventually had to do the entire development inside a 2G VirtualBox VM on my 8G Laptop.

The tricky part was that we had to submit the URL to the PoC Landing Page even before it could be ported to a full-fledged VM/server.

So effectively, we:
  • Submitted a URL (on ServerA) running Apache, but no space to host a 40G VM.
  • Had a Laptop where the VM was developed, but can't stay online since its not always connected
  • Had another server (ServerB) where the VM needs to be ported, to be always up for PoC evaluation
  • ServerA was in the US, whereas, ServerB (& Laptop) are half-way across the world in India.
Separately, the VM was an Ubuntu installation that had the following:
  • A running NodeJS (Bootstrap / Express combo) serving the PoC GUI
  • Separately it hosted the docker runtime that spun off Docker instances that had their own propietary application + Tomcat running on port 8080, all of which eventually supposed to be visible company-wide
  • Thirdly, it had a PostgreSQL 9.3 database that served as a backend to all the running docker-based app instances

To connect ServerA with the PoC hosted on ServerB we tried the following:
  • Creating VirtualHost entries on the ServerA that pointed to corresponding ports on the VM
    • This failed because the VM was running inside a NATted configuration and thus was invisible to ServerA
  • Then we tried an alternate solution, wherein Apache->VirtualHost entries on ServerA were pointed to port 80 on ServerB. This required port-forwarding port 80 on ServerB to port 3000 (NodeJS) on ServerB. Although for this we used NetSH, this failed for various reasons, some of which we could identify and some we couldn't. For e.g. we installed the ipv4 module for NetSH, also tried installing ipv6, as well as, enabling Windows Firewall for NetSH to work (contrary to popular belief that Windows Firewall might be the reason for things getting blocked)
  • We finally settled down to running persistent SSH connections directly from the VM to ServerA (thereby by-passing all Windows Firewall issues) and setting up Remote Port-Forward such that remote ports could connect to Node Server inside the VM
    • This too initially didn't work as expected. To identify the cause we:
      • isabled SELinux on ServerB, then ServerA (although that didn't make sense) as well as on the VM to no avail
      • Then we disabled IPTables on all the relevant machines and that still didn't help
      • Finally we realised that the Remote Port-Forwarding though working well, was getting attached to the localhost interface on ServerA. Which means that although 127.0.0.1:80 on ServerA was working as expected, 192.168.1.1:80 on the same server immediately gave a "connection failed". It took a while to realise that though I was logging in trans-atlantic, the immediate 'Connection Failed' was an issue with ServerA and not with this side of the line.
      • What complicated things post that was that the regular -g option was insufficient. It required setting the GatewayPorts on the SSHD (with a sshd restart) to work as expected.

Eventually, after a few hours of transatlantic jugglery, finally got a working URL that looked something like this... What a day !



1 comment:

Subba Reddy said...

Thanks for capturing it in a Blog post. Hope it comes in handy for someone.