Wednesday, December 04, 2013

Pydev defect with Django 1.6?

In Pydev 3.0 Django menu, you can start the shell with current project path included. However, when using this functionality with Django 1.6 project, the following error will be thrown:
from django.core import management;import tutorial.settings as settings;management.setup_environ(settings) Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'setup_environ'
It turned out that module setup_environ was removed from django.core.management in v1.6 while pydev still calls this module.
In order to fix the issue, you just need to run the following statement:
os.environ['DJANGO_SETTINGS_MODULE'] = '(project).settings'
Please replace (project) with your django project name.

Wednesday, November 20, 2013

ssh tunnel, a small but useful trick to bypass firewalls

I used to use ssh tunnel a lot during college times. It proved still very very useful in my work. It's not a trick that most developers know that's why I'm going to give a brief explanation here.

What usually happens during enterprise application development is that some services exposed over Internet are protected by firewalls, which are configured to only accept communications from certain servers via pre-defined ports.
Here's a real-world scenario I experience in mutiple projects:
1. Development team works on their desktops/laptops with a bunch of development tools. And they also have ssh access to development servers (Linux based).
2. There are some message queue services, or SMTP service exposed only to development servers.
3. Developers want to know what happens to the queue (to check whether the queue manager is working fine, and to do peek, enqueue, dequeue operations) but they cannot access the internet-based queue services from their desktops directly. Also it's not recommended (sometimes forbidden) to use GUI applications such as vnc to access development server. It's also a bit more complex for everyone to use shell script in development server to performance development activities.

In order for developers to access the service from their own desktops, only ssh tunnel needs to be configured. Local port forwarding does the work and makes sure to forward all communications of a pre-defined local port to a certain target via ssh tunnel.
In our scenario, suppose we use local port 9000 as the port to be forwarded. Suppose our dev server is named "devserver" and the remote queue service is on "qservice" with port 1515. We need to configure to forward communications of localhost:9000 to qservice:1515 via "devserver".

In practice, this can be done easily by putty with the following 2 steps:
1. Start putty and select the ssh session to access devserver. (Don't login yet)
2. In the putty left-side menu, select connection --> SSH --> Tunnels, then "Add new forwarded port" using the following value: Source port: 9000, Destination: qservice:1515, Local. After this, you can login to the dev server.
What above steps do is starting a local process which opens a ssh tunnel with devserver, and also listens to localhost port 9000, whenever their's any data received, it'll forward the data to qservice from the devserver via the ssh tunnel. As a result, developers can connect to local 9000 port to use the queue service.

Thursday, November 07, 2013

GAE application behaves abnormally after deployment

I've been playing with Flickr API with a GAE webapp.
 To maximize the performance, I didn't use some framework to build it. Basic APIs such as http connections are used to handle the communication.
The small app has been auto-tested thousands of times on local GAE run-time, and it's working fine totally. And no GAE datastore is used for this app (hence no datastore indexes need to be generated); I expect the app to work fine after a successful deployment.

However I faced the issue that 45% of the http request gave the same error message, complaining the invalid format of the data stream (returned from Flickr REST service). From the debug message trace, it turned out almost half of the requests, which are supposed to get data in JSON format (by setting JSON format as one of the query parameters), got routed to flickr API community guidelines page, resulting in that irrelevant data in html format was returned, although every time the HTTP GET was requested towards the same Flickr url.

I did some googling, and was not able to find out why this happened. I was thinking probably Flickr partially blocked/redirected requests from google (GAE) because it was a Yahoo company. Not sure about that. On the second day, I just tested the same app again. It works totally fine again, just as what I experienced with the local GAE run-time. It seems there's always a delay before a newly deployed GAE app to behave properly.
 This is a Java app; I'll check if Python app behaves similarly.

Friday, October 25, 2013

Variable placeholder conflict between AngularJs and Jinja2 template

It's quite common that in web page templates (diango or jinja2 templates) brackets "{{}}" are used as placeholders for template variables. When people want to use angularjs in those template files, angularjs might not work properly because it also uses "{{}}" to bind data model. A workaround for this is to redefine the templates; an example for Jinja2 template is as follows:
JINJA_ENVIRONMENT=jinja2.Environment( loader=jinja2.FileSystemLoader(os.path.dirname(__file__)), extensions=['jinja2.ext.autoescape'], variable_start_string='((', variable_end_string='))', autoescape=True)
After this step is done, in the template files, "(())" refers to template variables, and "{{}}" refers to AngularJs data model.

Monday, August 19, 2013

DGrid data refreshing event?

I'm working on dojo chart generation based on the data from dojo dgrid. 
The logic I'm building is as follows: 
1. Subscribe to a topic in the main page: 
connect.subscribe("dgridrefresh", mModule.generateRestChart); 
2. Initiate a JsonRest datastore. Use the datastore to create a dgrid with pagination plugin. 
3. Start the dgrid in some contentpane. 
4. Publish an event to a topic upon "dgrid-refresh-complete" event. 
The code is as follows (topicname is the paramter): 
dgrid.on("dgrid-refresh-complete", function(event) { 
                                console.log("refresh-complete. Publishing to " + topicname); 
                                require(["dojo/_base/connect"], function(connect) { 
                                        connect.publish(topicname, [event]); 
                                }); 
                        }); 

However, I observed that mModule.generateRestChart() function was only invoked once when the dgrid was initiated. I expect the method being called whenever pagination is executed. 

Same information is here:http://dojo-toolkit.33424.n3.nabble.com/get-data-refresh-notification-from-dgrid-with-jsonrest-datastore-td3998670.html

In the end, it turned out the dgrid-refresh-complete event will only be emitted once when dgrid is refreshed with loading the first page of data.
I did a workaround by customizing the pagination plugin for dgrid. The change I made was pretty simple: inside the Deferred block of gotoPage method of dgrid.extensions.Pagination, I added another emit of dgrid-navigation-complete event, which is a customized event by myself.

pointcuts before and after an ajax call in dojo?

I've been struggling in searching for proper pointcuts for dojo ajax calls.
The problem is that I'm using JsonRest with DGrid and pagination plugin -- actually it's not a "problem" because I chose to do so in order to shorten the development cycle for core frontend functionality. However, this combination doesn't offer the solution to several minor issues. For example, I want to show a loading picture, instead of just a loading message.
I decided to use aspect advice functions, which is a great piece of functionality available after dojo 1.7. It's not so easy to find the pointcuts to define advices. Reasons are here: http://dojo-toolkit.33424.n3.nabble.com/dojo-aspect-before-function-not-firing-td3985714.html#a3998683 )

In the end, I got some workaround to advise on _create method of dojo/request/xhr. But as I mentioned it's not a nice solution. Hopefully sitepen guru could expose some pointcuts in future releases.

Httpheaders to enable CORS filter for REST service used by dojo jsonrest data store

Dojo jsonrest datastore provides a simple way to implement ajax communication between dojo data grid (datagrid, enhancegrid, dgrid, or ondemandgrid) and the web service provider.
However, most times people would face CORS access issue when using jsonrest api.
Here's a few key points for the server side to enable the connection with jsonrest:
1. Access-Control-Allow-Origin, this header defines the allowed originate json request client. If not set, errors like "origin not allowed" will be thrown.
2. Access-Control-Allow-Headers, this header defines the allowed requesting headers, such as X-Requested-With, X-Range, Range, and so on. If not set, errors like "header not allowed" will be thrown.
3. Access-Control-Expose-Headers, this header defines the allowed response headers, such as Content-Range. If not set, error like "refused unsafe header" will be thrown.
X-Requested-With, X-Range, Range, Content-Range are the key headers attributes for jsonrest api to communciate with the server. They have to be enabled in the http header.

In order to enable them, we need to add/set a few headers from server side programmatically. Using java servlet/tomcat as an example, we can use server side filter to achieve this.
There's existing CORS filter library available at http://software.dzhuvinov.com/cors-filter-installation.html . However, it didn't work well for me. Implementing a custom CORS filter isn't that difficult as well. Here's the code:
public class CorsFilter implements Filter {

public CorsFilter() { }
public void init(FilterConfig fConfig) throws ServletException { }
public void destroy() { }
public void doFilter(
ServletRequest request, ServletResponse response, 
FilterChain chain) throws IOException, ServletException {
HttpServletResponse res=(HttpServletResponse)response;
res.addHeader("Access-Control-Allow-Origin", "*");
if(res.getHeader("Access-Control-Allow-Headers")==null) {
res.addHeader("Access-Control-Allow-Headers", "Origin, X-Requested-With, X-Range, Range, Content-Type, Accept");
}else{
res.setHeader("Access-Control-Allow-Headers", "Origin, X-Requested-With, X-Range, Range, Content-Type, Accept,"+res.getHeader("Access-Control-Allow-Headers"));
}
res.addHeader("Access-Control-Expose-Headers","Accept-Ranges, Content-Encoding, Content-Length, Content-Range");
chain.doFilter(request, response);
}
}
 After configuring this filter to apply to all requesting paths, it'll for sure enable those dojo jsonrest headers.
Please feel free to comment on whether there's easier ways to do this.

Tuesday, August 06, 2013

customize the format for dojo widgets

Recently I just started the POC on a small reporting tool based on dojo (The major effort is still at the server side - controller & model layer, which sends JSON objects to be presented in the frontend.)
After going through a few documentation on the dojotoolkit website, I still could not find the information how I can customize the look and feel of widget. Good thing about javascript library is that you can always see the code. -- after browsing the contents of the dijit/themes/claro subdirectories, it looks not so difficult for me to find out how to do the customization, because the dojo class names are quite intuitive.
For example, if I'd like to change the style of the accordion title, I can use the following selection in the css file (menuPanel is the container of the accordion in my html file):
#menuPanel .dijitAccordionTitle{
...
}
#menuPanel .dijitAccordionInnerContainerActive .dijitAccordionTitle, #menuPanel .dijitAccordionInnerContainerSelected .dijitAccordionTitle{
...
}
To change the title or tab label for tab container, the following can be used (centerPanel is something from my code, not part of dojo):
#centerPanel .dijitTabChecked {
...
}
#centerPanel .dijitTabContainerTop-tabs{
...
}
Things become easier if you dig a bit more :-)

Monday, July 29, 2013

Customizing dojo theme

Default dojo themes look quite boring. There's a quick way create your own themes by customizing existing ones. Here's the instructions to do so:
http://davidwalsh.name/dijit-theme

Friday, July 05, 2013

Enable static contents in Spring MVC

According to Spring MVC documentation http://static.springsource.org/spring/docs/3.2.2.RELEASE/spring-framework-reference/html/mvc.html#mvc-config-static-resources , adding the following bean configuration would do the work for DispatcherServlet serving static content:
 mapping="/resources/**" location="/public-resources/"/>
However, people tend to overlook the following 2 steps in order to make it completely work:
1. mvc:annotation-driven needs to be added as well. Otherwise, the above configuration would block DispatcherServlet to dispatch other dynamic content to controllers.
2. Make sure request path for those static content is correct, i.e., including context path.

Saturday, May 25, 2013

Apache 2.4, Django 1.5, Python 3.3 configuration on Windows

It's not so easy to get Django 1.5 configured with Apache 2.4 on windows.
Most information/documentation related to such configuration is on Linux and on earlier versions of apache or django. Some of them might be misleading for the configuration on windows. For example, some say privilege needs to be changed, others say wsgi.py file needs to be changed.
Here I'd like to explain what I've done to make it work on windows.
1. Get mod_wsgi of the correct version for apache and python. (Believe me it's not so easy to compile it by yourself on windows.) Where to get it is described in my previous post.
2. Configure the module loading in apache http server by first adding the module to apache module directory and then add "LoadModule wsgi_module modules/mod_wsgi.so" to the httpd.conf file.
3. Add "WSGIScriptAlias /django "D:\...\webprojects\tutorial\tutorial\wsgi.py"" to the configuration file. This basically says, for any request with path "django", please use wsgi.py to handle it.
4. Add "WSGIPythonPath "D:\...\webprojects\tutorial"" to the conf file. This points out where python modules can be found. You can add multiple paths (separated by ";") to it.
5. Add the following directory directive:
<Directory "D:\...\webprojects\tutorial">
<Files wsgi.py>
  Order deny,allow
  Allow from all
</Files>
</Directory>
This allows the access to all wsgi.py files in the directory and its subdirectories.
6. Description of all the above steps can be found with a bit googling. However, after all these steps are done properly, you would still get http 403 access denied error. (In the error log, you'll see "AH01630: client denied by server configuration:").
It seems the "Allow"  directive not working at all.
By checking http://httpd.apache.org/docs/2.4/mod/mod_access_compat.html#allow ; it can be seen "The directives provided by mod_access_compat have been deprecated by the new authz refactoring. Please see mod_authz_host." By following the link, you'll find that "Require" directive should be used instead.
So what still needs to be done is to replace "Allow from all" with "Require all granted".
After these 6 steps, you django 1.5 app can be running on apache 2.4 on windows.

Thursday, May 09, 2013

Configure Apache 2.4 with Django (Python3.3) on windows

Installing/configuring Apache2.4 with Django on Linux is easy, while doing the same thing on windows can be tricky due to incompatible vc libs, win sdk libs, python libs and so on. I've tried to compile the mod_wsgi several times, but faced some strange errors (I'm sure make file is updated properly). I found a good site where you can get compiled wsgi module: http://www.lfd.uci.edu/~gohlke/pythonlibs/#mod_wsgi .