Serving JSON data the JSONP way with Groovlets

If we want to serve JSON data and want it to be cross-domain accessible we can implement JSONP. This means that if we have a Groovlet with JSON output and want users to be able to access it with an AJAX request from the web browser we must implment JSONP. JSONP isn't difficult at all, but if we implement it our Groovlet is much more useful, because AJAX requests can be made from the web browser to our Groovlet.

The normal browser security model only allows calls to be made to the same domain as the web page from which the calls are made. One of the solutions to overcome this is the use of the script tag to load data. JSONP uses this method and basically let's the client decide on a bit of text to prepend the JSON data and enclose it in parentheses. This way our JSON data is encapsulated in a Javascript method and is valid to be loaded by the script element!

The following code shows a simple JSON data structure:

{ "title" : "Simple JSON Data", 
  "items" : [ 
    { "source" : "document", "author" : "mrhaki" }
    { "source" : "web", "author" : "unknown"}
  ]
}

If the client decides to use the text jsontest19201 to make it JSONP we get:

jsontest19201({ "title" : "Simple JSON Data", 
  "items" : [ 
    { "source" : "document", "author" : "mrhaki" }
    { "source" : "web", "author" : "unknown"}
  ]
})

Okay, so what do we need to have this in our Groovlet? The request for the Groovlet needs to be extended with a query parameter. The value of this query parameter is the text the user decided on to encapsulate the JSON data in. We will use the query parameter callback or jsonp to get the text and prepend it to the JSON data (notice we use Json-lib to create the JSON data):

import net.sf.json.JSONObject

response.setContentType('application/json')

def jsonOutput = JSONObject.fromObject([title: 'Simple JSON data'])
jsonOutput.accumulate('items', [source: 'document', author: 'mrhaki'])
jsonOutput.accumulate('items', [source: 'web', author: 'unknown'])

// Check query parameters callback or jsonp (just wanted to show off
// the Elvis operator - so we have two query parameters)
def jsonp = params.callback ?: params.jsonp
if (jsonp) print jsonp + '('
jsonOutput.write(out)
if (jsonp) print ')'

We deploy this Groovlet to our server. For this blog post I've uploaded the Groovlet to Google App Engine. The complete URL is http://mrhakis.appspot.com/jsonpsample.groovy. So if we get this URL without any query parameters we get:

{"title":"Simple JSON data","items":[{"source":"document","author":"mrhaki"},{"source":"web","author":"unknown"}]}

Now we get this URL again but append the query parameter callback=jsontest90210 (http://mrhakis.appspot.com/jsonpsample.groovy?callback=jsontest90210) and get the following output:

jsontest90210({"title":"Simple JSON data","items":[{"source":"document","author":"mrhaki"},{"source":"web","author":"unknown"}]})

We would have gotten the same result if we used http://mrhakis.appspot.com/jsonpsample.groovy?jsonp=jsontest90210. The good thing is user's can now use for example jQuery's getJSON() method to get the results from our Groovlet from any web page served on any domain.

The following is generated with jQuery.getJSON() and the following code:

$(document).ready(function() {
  $.getJSON('http://mrhakis.appspot.com/jsonpsample.groovy?callback=?', function(data) {
    $.each(data.items, function(i, item) {
      $("<p/>").text("json says: " + item.source + " - " + item.author).appendTo("#jsonsampleitems");
    });
  });
});

JSONP output:

Add Client Side Caching to Groovlets and Separate Logic and View

We are going to see how we can implement client side caching for our Groovlet we made in the previous blog. Client side caching decreases bandwidth usage and also lessens the load on our application. Besides a faster response for the users we can even save costs if we have to pay for bandwidth usage if the appliation is running in a cloud for example. We will also refactor the Groovlet and use a view template for displaying the data and remove all HTML generation from the Groovlet. This way we have a better separation between the code and the HTML.

The current Groovlet display a list of activities and each activity has a certain date. We use this date to print out nice human readable, relative timestamps like we see on Twitter. For example 2 days ago, just now, 1 week ago. Each time we refresh the Groovlet we want to update this information. Because the time is relative compared to the current date and time we cannot cache this data. Even if all activity information stays the same we still have to update the human readable timestamps, otherwise the timestamps don't make sense. To solve this and make the data cachable we move the code to generate the human readable timestamps from the server to the client. On the client side we use Javascript to generate the relative date and time values. Now we can implement client side caching for our Groovlet output, because the uncachable date and time values are generated on the client side and not on the server.

Here is the code for the Groovlet. We place it in a file war/WEB-INF/groovy/viewactivities.groovy:

import net.sf.json.groovy.JsonSlurper
import net.sf.json.JSONException
import org.apache.commons.codec.digest.DigestUtils

// We keep track of the last time the content of the Yahoo! Pipe
// has changed in the application context.
now = new Date()
lastUpdated = application.getAttribute('lastUpdated')
if (!lastUpdated) {
    lastUpdated = now
    application.setAttribute('lastUpdated', lastUpdated)
}

// Get the contents of the Yahoo! Pipe.
pipeUrl = 'http://pipes.yahoo.com/pipes/pipe.run?_id=UtVVPkx83hGZ2wKUKX1_0w&_render=json'
result = urlFetchService.fetch(new URL(pipeUrl))
def resultString = new String(result.content)

// The Yahoo! Pipe contains a pubDate element, which contains the date and time
// the Pipe is requested. This changes each time we make a request, even all other
// data stays the same. We filter out this date, so we can make a MD5 hash without
// the dynamic date and time value for better comparison.
resultStringNoPubDate = resultString.replaceFirst(/"pubDate":"\w{3}, \d{2,} \w{3} \d{4} \d{2}:\d{2}:\d{2} [+-]\d{4}",/, '')

// Determine MD5 hash for the Pipe's content.
etag = DigestUtils.md5Hex(resultStringNoPubDate)

// If the request header contains the 'If-None-Match' header we must see if the 
// value is the same as we calculated above. If so the content of the Pipe has not changed
// and we can return a 304 Not Modified response.
// Or if the request header contains a 'If-Modified-Since' header we see if the value
// is the same as the last time the content has changed. We return a 304 Not Modified
// if this is the case.
// The response is send directly to the client without any further content.
if (headers['If-None-Match'] == etag || headers['If-Modified-Since'] == lastUpdated.time) {
    response.sendError javax.servlet.http.HttpServletResponse.SC_NOT_MODIFIED
    return
}

// The content has changed so we update the lastUpdated value.
lastUpdated = now
application.setAttribute('lastUpdated', lastUpdated)

// We set the response headers 'ETag', 'Last-modified'
// and 'Cache-Control'. Cache-control is set to 5 minutes.
response.setHeader('ETag', etag)
response.setDateHeader('Last-modified', lastUpdated.time)
response.setHeader('Cache-Control', 'max-age=' + 5 * 60)

// Set request attributes with values we can use in the 'activities.gtpl' view.
if (resultString) {
    try {
        def jsonReader = new JsonSlurper()
        def json = jsonReader.parseText(resultString)
        request.setAttribute('json', json)

    } catch (JSONException jsonException) {
        request.setAttribute('error', jsonException.getMessage() + "
${resultString}
")
    }
} else {
    request.setAttribute('error', 'No results found. Try again later.')
}

// Go to the view 'activities.gtpl'.
forward 'activities.gtpl' 

At line 22 we remove the value for the pubDate element from the Yahoo! Pipe. This value is updated each time we make a request. If we want to be able to compare the MD5 hash of the content from a previous request with the current result we must exclude the dynamic date and time value. If we don't remove it we always get a different MD5 hash value and we wouldn't be able to send back a 304 Not Modified response.

In line 25 we calculate the MD5 hash with the Apache Commons Codec library. We need to place the JAR file in the war/WEB-INF/lib directory of our Google App Engine application.

At line 34 we see if the request header If-None-Match is available. The value of this header is the MD5 hash of the content from a previous request. We compare this value to the freshly calculated value in line 25. If they are equal we know the content of the Pipe hasn't changed, so the client can use a cached version of the page. As a fallback we also check the If-Modified-Since header in case the client doesn't support If-None-Match.

We set the response header Cache-Control to five minutes at line 47. This means the client can use a cached copy of the page for five minutes before it needs to check the server again to see if something has changed. This means if the content has changed in five minutes after the last request, the client will not see it. For this application that is acceptable, but we always need to check if such a period is acceptable or not for the users.

At lines 50-61 we populate the request with attributes we can use in the view template. The JSON results is stored in a request variable named json. If any error has occured we save it in the request variable error.

At the last line we leave the Groovlet and continue on to the view activites.gtpl.

We create a view activites.gtpl in the directory war to display the results:

<%
import java.text.SimpleDateFormat
import java.text.ParseException

// Closure to parse the date string from the activities.
def parseActivityDate = { dateString ->
    def result
    dateString = dateString.replaceAll(/(\d{2}):(\d{2})$/, '$1$2')
    def parsers = [ 
        new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss Z", Locale.US),
        new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'", Locale.US),
        new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ", Locale.US)
    ]
    for (SimpleDateFormat parser : parsers) {
        try {
            result = parser.parse(dateString)
            break
        } catch (ParseException e) {
            continue
        }
    }
    result        
}
%>

<html>
<head>
    <title>Social Activities</title>
    <link type='text/css' rel='stylesheet' href='css/activities.css'/>
</head>
<body>
    <h1>Activities</h1>

    <% if (request.getAttribute('error')) { %>
      ${request.getAttribute('error')
    <% } else { %>
      <div id="activities">

      <% request.getAttribute('json').value.items.each { item -> %>
        <div class="item">
          <img src="images/socialicons/32x32/${item.source}.png" title="${item.source}" alt="${item.source}"/>
          <a href="${item.link}">${item.title}</a>
          <br />
          <span class="date" title="${parseActivityDate(item.pubDate).format("yyyy-MM-dd'T'HH:mm:ss")}"> </span>
      </div>
    <% } %>
    </div>
  
    <script src='js/jquery.js' type='text/javascript'> </script>
    <script src='js/humane.js' type='text/javascript'> </script>
    <script type='text/javascript'>
     \$(document).ready(function() {
       \$("span.date").humane_dates();
       setInterval(function() {
        \$("span.date").humane_dates();
       }, 5000);
    });
    </script>
   <% } %>

</body>
</html>

At the top we define a closure to parse the date strings from the JSON results. We use it at line 43 to format the date for the title attribute of the span element. At line 50 we import the Yet Another Pretty Date Javascript library. This library will use the value of the title attribute of an element and create a human readable timestamp. At lines 52-57 we define Javascript to calculate the timestamp values and display them on the page. We also define an interval of five seconds to update the information. This way the information is always up-to-date without refreshing the page. Notice how we must escape the $ sign to have it placed literally in the output, otherwise Groovy thinks it is an Groovy expression.

The rest of the code is straightforward. At line 38 we loop through the JSON results and display a div element with the information for each item.

We have seen how we can add reponse headers to our Groovlet to support client side caching. We set the Cache-Control header to minimize the number of requests to the server. And if a request hits our Groovlet we check if the contents has changed. If there is no change we return a 304 Not Modified response, so a cached version on the client side can be used. Furthermore we refactored the previous Groovlet to separate the logic from the view. A more spiced up example of the application is available on Google App Engine.

'Invalid label' when using jQuery.getJSON() method

I was working on a small project and I wanted to test some Javascript code that read data formatted as JSON. I setup a small HTML file, sample.html, with a reference to the jQuery library and invoke $.getJSON("http://localhost/example.json", function(data) { alert(data); });. In Firefox I opened http://localhost/sample.html and suspected to see some data. But instead I got an Invalid label in Firebug. What happened? After reading this Stackoverflow answer I knew why. I invoked the $.getJSON() method with a URL starting with http://, and jQuery automatically assumes the request is a JSONP request. But in this case that is not true. Replacing the URL with the relative example.json is the solution.

Using jQuery's JSONP support to get data from Yahoo! Pipe

In a previous blog entry we learned how to create a Yahoo! Pipe. In this blog entry we will see how we can use jQuery to parse the Yahoo! Pipe's output.

When we run our pipe we have a link Get as JSON. We need the URL of this link so we can use it with our jQuery script. Basically it is the URL of the pipe appended with &_render=json.

Normally we cannot get data from just any domain on the internet with Javascript. But with JSONP we can get data from other domains. jQuery and Yahoo! Pipes supports JSONP, we only have to add &_callback=? to our Yahoo! Pipe JSON URL and we are in business. We use the $.getJSON method to get the data from our pipe. Once we get hold of the data we can use it any way we want. The following code snippet loads the data and then creates a list with the title, link, date and source of the items:

$(document).ready(function() {
    $.getJSON("http://pipes.yahoo.com/pipes/pipe.run?_id=UtVVPkx83hGZ2wKUKX1_0w&_render=json&_callback=?", 
    function(data) {
        $.each(data.value.items, function(idx, item) {
            var listitem = $("<li/>");
            $("<a/>").attr("href", item.link).attr("class", item.source).text(item.title).appendTo(listitem);
            $("<br />").appendTo(listitem);
            $("<span/>").attr("class", "source").text("from " + item.source).appendTo(listitem);
            $("<span/>").text(", " + item.pubDate).appendTo(listitem);
            listitem.appendTo("#activities");  // append to UL element with id activities
        });
    });
});

All put together in a HTML page and a little styling we get the following output:

And here is the complete HTML source:

<html>
<head>
  <script src="http://code.jquery.com/jquery-latest.js"></script>

  <script>
  $(document).ready(function() {
    $.getJSON("http://pipes.yahoo.com/pipes/pipe.run?_id=UtVVPkx83hGZ2wKUKX1_0w&_render=json&_callback=?",
        function(data) {
          $.each(data.value.items, function(idx,item) {
            var listitem = $("<li/>");
            $("<a/>").attr("href", item.link).attr("class", item.source).text(item.title).appendTo(listitem);
            $("<br />").appendTo(listitem);
            $("<span/>").attr("class", "source").text("from " + item.source).appendTo(listitem);
            $("<span/>").text(", " + item.pubDate).appendTo(listitem);
            listitem.appendTo("#activities");
          });
        });
  });
  </script>
  <style>
    .source { font-style: italic; }
  </style>
</head>
<body>
  <h1>Yahoo! Pipes</h1>
  <ul id="activities">
  </ul>
</body>
</html>