Wednesday, November 20, 2013

ssh tunnel, a small but useful trick to bypass firewalls

I used to use ssh tunnel a lot during college times. It proved still very very useful in my work. It's not a trick that most developers know that's why I'm going to give a brief explanation here.

What usually happens during enterprise application development is that some services exposed over Internet are protected by firewalls, which are configured to only accept communications from certain servers via pre-defined ports.
Here's a real-world scenario I experience in mutiple projects:
1. Development team works on their desktops/laptops with a bunch of development tools. And they also have ssh access to development servers (Linux based).
2. There are some message queue services, or SMTP service exposed only to development servers.
3. Developers want to know what happens to the queue (to check whether the queue manager is working fine, and to do peek, enqueue, dequeue operations) but they cannot access the internet-based queue services from their desktops directly. Also it's not recommended (sometimes forbidden) to use GUI applications such as vnc to access development server. It's also a bit more complex for everyone to use shell script in development server to performance development activities.

In order for developers to access the service from their own desktops, only ssh tunnel needs to be configured. Local port forwarding does the work and makes sure to forward all communications of a pre-defined local port to a certain target via ssh tunnel.
In our scenario, suppose we use local port 9000 as the port to be forwarded. Suppose our dev server is named "devserver" and the remote queue service is on "qservice" with port 1515. We need to configure to forward communications of localhost:9000 to qservice:1515 via "devserver".

In practice, this can be done easily by putty with the following 2 steps:
1. Start putty and select the ssh session to access devserver. (Don't login yet)
2. In the putty left-side menu, select connection --> SSH --> Tunnels, then "Add new forwarded port" using the following value: Source port: 9000, Destination: qservice:1515, Local. After this, you can login to the dev server.
What above steps do is starting a local process which opens a ssh tunnel with devserver, and also listens to localhost port 9000, whenever their's any data received, it'll forward the data to qservice from the devserver via the ssh tunnel. As a result, developers can connect to local 9000 port to use the queue service.

Thursday, November 07, 2013

GAE application behaves abnormally after deployment

I've been playing with Flickr API with a GAE webapp.
 To maximize the performance, I didn't use some framework to build it. Basic APIs such as http connections are used to handle the communication.
The small app has been auto-tested thousands of times on local GAE run-time, and it's working fine totally. And no GAE datastore is used for this app (hence no datastore indexes need to be generated); I expect the app to work fine after a successful deployment.

However I faced the issue that 45% of the http request gave the same error message, complaining the invalid format of the data stream (returned from Flickr REST service). From the debug message trace, it turned out almost half of the requests, which are supposed to get data in JSON format (by setting JSON format as one of the query parameters), got routed to flickr API community guidelines page, resulting in that irrelevant data in html format was returned, although every time the HTTP GET was requested towards the same Flickr url.

I did some googling, and was not able to find out why this happened. I was thinking probably Flickr partially blocked/redirected requests from google (GAE) because it was a Yahoo company. Not sure about that. On the second day, I just tested the same app again. It works totally fine again, just as what I experienced with the local GAE run-time. It seems there's always a delay before a newly deployed GAE app to behave properly.
 This is a Java app; I'll check if Python app behaves similarly.