A Web Coding Blog

Why are JSPs so slow? (Tomcat 7 vs. Apache)

James H. (admin) — Tue, 21 Jun 2011 20:02:15 +0000

It would be unreasonable to expect JSPs to be served at the same rate as static files. Even after being compiled, requests to JSPs must pass through layers of servlet container code, not to mention the latency of the JSP code itself. However, the difference between static files vs. JSP is much larger than I would have expected.

I ran a test on a 6KB HTML file using Apache Benchmark on my Dell Inspiron N7010 laptop, using 1,000 requests @ 10 concurrent requests. I also created a JSP by pasting the same HTML into a file, adding only a few dynamic elements: outputting the of the host name, port, and app URI. No calls to any beans, JDBC, or other potentially slow resources were made. Default installations were used for both Apache and Tomcat 7. The results were stark:

Apache static HTML file: 971.25 pages/second (blazin’)
Tomcat 7 JSP file: 92.5 pages/second

As you can see, the JSP is not merely slower. It is a full order of magnitude slower. Because the dynamic content is trivial in this test file, the performance of the servlet container fully accounts for this difference.

I thought I’d also test loading the JSP through Apache using mod_jk to see what kind of performance penalty this imparts. Results:

Tomcat 7 JSP file via Apache mod_jk: 74.80 pages/second

So in this test case, there was a 19% performance penalty fronting Tomcat with Apache. Of course speed is not the only consideration when evaluating using this type of setup, but it is something to weigh against other concerns.

There is also a debate about whether Tomcat or Apache is faster at serving static files. This test is only one data point, but Apache easily wins here. Tomcat serving the static 6KB file:

Tomcat 7 static HTML file: 203.5 pages/second

In this test, Apache was nearly five times faster at serving this file (though more than twice as fast compared to the JSP version).

And now, for people who like charts. =)

Configure Lucene IndexWriter and IndexSearcher in Spring applicationContext.xml

James H. (admin) — Wed, 01 Jun 2011 17:13:06 +0000

Problem: you want to define Lucene IndexWriter and IndexSearcher as beans inside your Spring application to be injected/autowired into other beans.

Solution: follow the following steps.

define the Lucene version as a constant
define a Lucene analyzer (StandardAnalyzer) as a bean
define a Lucene directory as a bean, using a factory-method for instantiation
define an IndexWriter, wiring in the Lucene directory and an IndexWriterConfig set to use your previously-defined analyzer
define an IndexSearcher, wiring in the Lucene directory
define also a query parser (StandardQueryParser), wiring in the analyzer bean

You can then wire/autowire these beans into your application beans, for example:

@Autowired
private IndexWriter indexWriter;

Here is the resulting XML config after following the above steps, which you will place in your Spring applicationContext.xml:

Now you’re ready to rock and roll with Lucene in your application. There is one problem, however. On first execution, instantiation of the IndexSearcher will fail because no index exists yet. I have not found an elegant solution to this problem, so what I do is simply comment out the IndexSearcher bean definition on first run. Doing so necessitates this bean be optional in your application. So, where I autowire IndexWriter, I set required=false:

@Autowired(required=false)
private IndexSearcher indexSearcher;

Once a Lucene index exists, I uncomment the searcher bean. If there is a better way to solve this problem, please share a comment!

Simple pagination taglib for JSP

James H. (admin) — Fri, 27 May 2011 17:39:08 +0000

Pagination is a common requirement when writing JSPs. Long sets of data must be broken up across multiple pages. There is no standard way to implement pagination in JSP or JSTL, however. You must use a custom tag to generate the page numbers.

I’ve created a simple pagination taglib that generates the page links. You can customize it to your needs. Usage example below. Here, I’m using it to paginate search results from Lucene:

The paginator:display tag produces this output:

This is obviously showing Page 4 of the search results. As you can see, you specify the URI, and the tag expects a ## where the page number will go. You also specify the current page, the total number of pages, and the maximum number of page links you want to show.

The paginator output is an unordered list, which you’ll want to style something like this:

.paginatorList { margin: 2px 6px; list-style: none outside none; }
.paginatorList li { float: left; padding: 2px 4px; font-size: 1.2em; }
li.paginatorCurr { font-weight: bold; font-size: 1.5em; margin-top: -2px; }
li.paginatorLast { float: none; }

This CSS will style the page links just like the example above.

Here is the java code for the paginator taglib:

package my.taglibs;

import java.io.Writer;
import javax.servlet.jsp.JspWriter;
import javax.servlet.jsp.JspException;
import javax.servlet.jsp.tagext.SimpleTagSupport;

public class Paginator extends SimpleTagSupport {
    private String uri;
    private int currPage;
    private int totalPages;
    private int maxLinks = 10;

    private Writer getWriter() {
        JspWriter out = getJspContext().getOut();
        return out;
    }

    @Override
    public void doTag() throws JspException {
        Writer out = getWriter();

        boolean lastPage = currPage == totalPages;
        int pgStart = Math.max(currPage - maxLinks / 2, 1);
        int pgEnd = pgStart + maxLinks;
        if (pgEnd > totalPages + 1) {
            int diff = pgEnd - totalPages;
            pgStart -= diff - 1;
            if (pgStart < 1)
                pgStart = 1;
            pgEnd = totalPages + 1;
        }

        try {
            out.write("");

            if (currPage > 1)
                out.write(constructLink(currPage - 1, "Previous", "paginatorPrev"));

            for (int i = pgStart; i < pgEnd; i++) {
                if (i == currPage)
                    out.write(""+ currPage + "");
                else
                    out.write(constructLink(i));
            }

            if (!lastPage)
                out.write(constructLink(currPage + 1, "Next", "paginatorNext paginatorLast"));

            out.write("");

        } catch (java.io.IOException ex) {
            throw new JspException("Error in Paginator tag", ex);
        }
    }

    private String constructLink(int page) {
        return constructLink(page, String.valueOf(page), null);
    }

    private String constructLink(int page, String text, String className) {
        StringBuilder link = new StringBuilder("")
            .append("")
            .append(text)
            .append("");
        return link.toString();
    }

    public void setUri(String uri) {
        this.uri = uri;
    }

    public void setCurrPage(int currPage) {
        this.currPage = currPage;
    }

    public void setTotalPages(int totalPages) {
        this.totalPages = totalPages;
    }

    public void setMaxLinks(int maxLinks) {
        this.maxLinks = maxLinks;
    }
}

And the .tld file:



  1.0
  paginator
  /WEB-INF/tlds/Paginator

  
    display
    my.taglibs.Paginator
    empty
    
      uri
      true
      true
      java.lang.String
    
    
      currPage
      true
      true
      int
    
    
      totalPages
      true
      true
      int
    
    
      maxLinks
      true
      int

Now you can easily add pagination to any JSP. Leave a comment if you found this useful! If you have a blog as well, I wouldn’t mind a link. ^_^

Convert between Java enums and PostgreSQL enums

James H. (admin) — Sun, 15 May 2011 16:48:12 +0000

PostgreSQL allows you to create enum types using the following syntax:

CREATE TYPE animal_type AS ENUM('DOG', 'CAT', 'SQUIRREL');

You can now use ‘animal’ as a datatype in your tables, for example:

create table pet (
                  pet_id        integer         not null,
                  pet_type      animal_type     not null,
                  name          varchar(20)     not null
                  );

In Java, you’d have a corresponding enum type:

public enum AnimalType {
    DOG,
    CAT,
    SQUIRREL;
 }

Converting between Java and PostgreSQL enums is straightforward. For example, to insert or update an enum field you could use the CAST syntax in your SQL PreparedStatement:

INSERT INTO pet (pet_id, pet_type, name) VALUES (?, CAST(? AS animal_type), ?);

--or

INSERT INTO pet (pet_id, pet_type, name) VALUES (?, ?::animal_type, ?);

Postgres will also let you insert/update an enum just by passing its value as a string.

Whether casting or not, the Java side is the same. You would set the fields like this:

stmt.setInt(1, 1);
stmt.setString(2, AnimalType.DOG.toString());
stmt.setString(3, 'Rex');

Retrieving the enum from a SELECT statement looks like this:

AnimalType.valueOf(stmt.getString("pet_type"));

Take into consideration that enums are case-sensitive, so any case mismatches between your Postgres enums and Java enums will have to be accounted for. Also note that the PostgreSQL enum type is non-standard SQL, and thus not portable.

Also, FYI, to view the set of values in a given Postgres enum type, you can use the following SQL query:

SELECT enumlabel FROM pg_enum
    WHERE enumtypid = 'your_enum'::regtype ORDER BY oid;

Web scraping in Java with Jsoup, Part 2 (How-to)

James H. (admin) — Sat, 14 May 2011 15:17:18 +0000

Web scraping refers to programmatically downloading a page and traversing its DOM to extract the data you are interested in. I wrote a parser class in Java to perform the web scraping for my blog analyzer project. In Part 1 of this how-to I explained how I set up the calling mechanism for executing the parser against blog URLs. Here, I explain the parser class itself.

But before getting into the code, it is important to take note of the HTML structure of the document that will be parsed. The pages of The Dish are quite heavy–full of menus and javascript and other stuff, but the area of interest is the set of blog posts themselves. This example shows the HTML structure of each blog post on The Dish:


    
        
            
                email
            
            
                permalink
            
            
            
        

        12 May 2011 07:37 PM
    

    
        
            Face Of The Day
        
        
            
                
            
        
        
        A decorated  veteran takes part [truncated]

Blog posts are each contained within an HTML5 article tag. There is a time tag holding the date and time the post was published. A div with class aentry holds both the title and body of the post. The title is within an h1 and also contains the permalink for the post.

Now, the code to parse this page.

The simple blog parser interface again:

public interface BlogParser {
    public List parseURL(URL url) throws ParseException;
}

Now to talk about the implementation class: DishBlogParser. The goal is to return a list of Link objects (a “Link” in this context represents one blog URL and its associated data). DishBlogParser will extract the title and body text of each blog post along with the post date, images, videos, and links contained therein. I’ll go through the class a section at a time. Starting from the top:

@Component("blogParser")
public class DishBlogParser implements BlogParser {

    @Value("${config.excerptLength}")
    private int excerptLength;
    @Autowired
    private DateTimeFormatter blogDateFormat;
    private final Cleaner cleaner;
    private final UrlValidator urlvalidator;

    public DishBlogParser() {
        Whitelist clean = Whitelist.simpleText().addTags("blockquote", "cite", "code", "p", "q", "s", "strike");
        cleaner = new Cleaner(clean);
        urlvalidator = new UrlValidator(new String[]{"http","https"});
    }

The excerptLength field defines the maximum length for post body excerpts. The @Value annotation pulls in the value from a properties file configured in applicationContext.xml.

The blogDateFormat is a Joda formatter configured also in applicationContext.xml to match the date/time format used on The Dish. It will be used to parse dates from HTML into Joda DateTime objects. Here is how blogDateFormat is configured in applicationContext.xml:

The Cleaner object is a Jsoup class that applies a whitelist filter to HTML. In this case, the cleaner is used to whitelist tags that will be allowed to appear in blog body excerpts.

Finally, the UrlValidator comes from Apache Commons and will be used to validate the syntax of URLs contained within blog posts.

Now, for the parseURL method:

    public List parseURL(URL url) throws ParseException {
        try {
            // retrieve the document using Jsoup
            Connection conn = Jsoup.connect(url.toString());
            conn.timeout(12000);
            conn.userAgent("Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)");
            Document doc = conn.get();

            // select all article tags
            Elements posts = doc.select("article");

            // base URI will be used within the loop below
            String baseUri = (new StringBuilder())
                .append(url.getProtocol())
                .append("://")
                .append(url.getHost())
                .toString();

            // initialize a list of Links
            List links = new ArrayList();

Here, Jsoup is used to connect to the URL. I set a generous connection timeout, because at times The Dish server is not very snappy. I also set a common user agent, just as a general practice when requesting a web page programmatically.

On Line 7 the Document is retrieved–this is a DOM representation of the entire page. For this project, only the blog posts themselves are needed. Because each blog post is contained in an article tag, the set of posts is obtained by calling doc.select("article") (Line 10). We’re about to loop through them, but first we need to define the base URI of our URL for something a bit further down, and also initialize the List which will hold our extracted Link objects.

Now, the loop. It starts like this:

            // loop through, extracting relevant data
            for (Element post : posts) {
                Link link = new Link();

                // extract the title of the post
                Elements elms = post.select(".entry h1");
                String title = (elms.isEmpty() ? "No Title" : elms.first().text().trim());
                link.setTitle(title);

First, an empty Link object is initialized. Then we extract the title. Recall that “post” is a Jsoup element pointing to the article tag in the DOM. post.select(".entry h1") grabs the h1 title tag, from which we get the title string.

In a similar fashion, we grab the URL and the date:

                // extract the URL of the post
                elms = post.select("aside .entryLink a");
                if (elms.isEmpty()) {
                    Logger.getLogger(DishBlogParser.class.getName()).log(Level.WARNING, "UNABLE TO LOCATE PERMALINK, TITLE = "+ title +", URL = "+ url);
                    continue;
                }
                link.setUrl(elms.first().attr("href"));

                // extract the date of the post
                elms = post.select("aside time");
                if (elms.isEmpty()) {
                    Logger.getLogger(DishBlogParser.class.getName()).log(Level.WARNING, "UNABLE TO LOCATE DATE, TITLE = "+ title +", URL = "+ url);
                    continue;
                }
                // parse the date string into a Joda DateTime object
                DateTime dt = blogDateFormat.parseDateTime(elms.first().text().trim());
                link.setLinkDate(dt);

Note that failure to extract the URL or date is unacceptable, a warning is logged, and further processing is skipped. Note also on Line 16 blogDateFormat is used to parse the date string from the HTML into a DateTime object.

Next, let’s grab the body of the post and create an excerpt from it:

                // extract the body of the post (includes title tag at this point)
                Elements body = post.select(".entry");
                // remove the "more" link
                body.select(".moreLink").remove();

                // remove the title (h1) now from the body
                body.select("h1").remove();
                // set full text on link, used for indexing/searching (not stored)
                link.setFullText(body.text());

                // create a body "Document"
                Document bodyDoc = Document.createShell(baseUri);
                for (Element bodyEl : body)
                    bodyDoc.body().appendChild(bodyEl);
                // remove unwanted tags by applying a tag whitelist
                // the whitelisted tags will appear when displaying excerpts
                String bodyhtml = cleaner.clean(bodyDoc).body().html();

                if (bodyhtml.length() > excerptLength) {
                    // we need to trim it down to excerptLength
                    bodyhtml = trimExerpt(bodyhtml, excerptLength);
                    // we need to parse this again now to fix possible unclosed tags caused by trimming
                    bodyhtml = Jsoup.parseBodyFragment(bodyhtml).body().html();
                }
                link.setExerpt(bodyhtml);

Recall the body is contained in a div classed entry. The body may contain a “read on” link that expands the content. That link, if present, is removed on Line 4. The title h1 tag is also removed, and the remaining text is stored on Line 9. This full text is not destined to be stored in the database–instead it will be indexed by our search engine.

To create the excerpt, unwanted HTML tags must be removed. This is where the Jsoup Cleaner comes in. Because the Cleaner only processes Document objects, a dummy Document is created for the post (this is also where baseUri is used).

If, after processing the post body through the Cleaner, the length exceeds the excerptLength, it must be trimmed down to size. The trimExcerpt method does this. Because trimming might truncate closing HTML tags, Jsoup is used once more to parse the excerpt string, correcting any unbalanced tags. Finally, we have our excerpt.

This is the trimExerpt method that is called on Line 21 above:

    private String trimExcerpt(String str, int maxLen) {
        if (str.length() <= maxLen)
            return str;

        int endIdx = maxLen;
        while (endIdx > 0 && str.charAt(endIdx) != ' ')
            endIdx--;

        return str.substring(0, endIdx);
    }

The idea is to use maxLen as a suggestion, and keep backing up until a space character is found. In this way, words will not be cut off in the middle.

Continuing the loop, next the links are extracted. They are represented by InnerLink objects. Any invalid or self links are skipped.

                // extract the links within the post
                List inlinks = new ArrayList();
                Elements innerlinks = body.select("a[href]");                

                // loop through each link, discarding self-links and invalids
                for (Element innerlink : innerlinks) {
                    String linkUrl = innerlink.attr("abs:href").trim();
                    if (linkUrl.equals(link.getUrl()))
                        continue;
                    else if (urlvalidator.isValid(linkUrl)) {
                        //System.out.println("link = "+ linkUrl);
                        InnerLink inlink = new InnerLink();
                        inlink.setUrl(linkUrl);
                        inlinks.add(inlink);
                    }
                    else
                        Logger.getLogger(DishBlogParser.class.getName()).log(Level.INFO, "INVALID URL: "+ linkUrl);
                }
                link.setInnerLinks(inlinks);

Next, extract any images:

                // extract the images from the post
                List linkimgs = new ArrayList();
                Elements images = body.select("img");
                for (Element image : images) {
                    Image img = new Image();
                    img.setOrigUrl(image.attr("src"));
                    img.setAltText(image.attr("alt").replaceAll("_", " "));
                    linkimgs.add(img);
                }
                link.setImages(linkimgs);

Finally, extract any Youtube or Vimeo videos (the two most-popular types). Note that this requires a more complex selector syntax (Line 2), in particular because over the years several different HTML codes have been used:

                // extract Youtube and Vimeo videos from the post
                elms = body.select("iframe[src~=(youtube\\.com|vimeo\\.com)], object[data~=(youtube\\.com|vimeo\\.com)], embed[src~=(youtube\\.com|vimeo\\.com)]");
                List videos = new ArrayList(2);
                for (Element video : elms) {
                    String vidurl = video.attr("src");
                    if (vidurl == null)
                        vidurl = video.attr("data");
                    if (vidurl == null || vidurl.trim().equals(""))
                        continue;
                    Video vid = new Video();
                    vid.setUrl(vidurl);
                    if (vidurl.toLowerCase().contains("vimeo.com"))
                        vid.setProvider(VideoProvider.VIMEO);
                    else
                        vid.setProvider(VideoProvider.YOUTUBE);
                    videos.add(vid);
                }
                link.setVideos(videos);

Finally, the loop is finished; all data has been gathered. So this Link object is added to the List, end loop, and return:

                links.add(link);
            }
            return links;
        }
        catch (IOException ex) {
            Logger.getLogger(DishBlogParser.class.getName()).log(Level.SEVERE, "IOException when attemping to parse URL "+ url, ex);
            throw new ParseException(ex);
        }
    }

In conclusion…

This post has demonstrated web scraping using the open-source Jsoup library. Specifically, we loaded a page from a URL and used Jsoup’s selector syntax to extract the desired pieces of data. In a future post, I will write about what happens next: the list of Links is processed by a service bean and stored in the database.

Disable URL session IDs (JSESSIONID) in Tomcat 7, Glassfish v3

James H. (admin) — Thu, 12 May 2011 15:53:57 +0000

URL-based session tracking is intended for web clients that do not support session cookies. Every browser worth mentioning supports these cookies, and almost nobody surfs with them disabled. Most web sites either state explicitly or assume that a user’s browser supports session cookies. URL rewriting schemes that add the session ID as a parameter on every URL thus provide very little benefit, if any at all. Session IDs showing up in URLs is just bad form, and may confuse search engine spiders. Thankfully the Servlet 3.0 standard gives you two ways to disable URL session rewriting. This works in Tomcat 7, Glassfish v3, and any other Servlet 3.0-compliant servlet container.

First, you can add this to your web.xml web-app config:


    COOKIE

Or programmatically, you can use:

servletContext.setSessionTrackingModes(EnumSet.of(SessionTrackingMode.COOKIE));

I’ve used the web.xml method in Tomcat 7, and it works. No jsessionid in the URLs when using in my JSPs.

Web scraping in Java with Jsoup, Part 1

James H. (admin) — Thu, 28 Apr 2011 18:55:22 +0000

In order to obtain the data to feed into my blog analyzer, content must be parsed from the pages of the blog itself. This is called “web scraping”. Jsoup will be used to parse the pages, and because this is a Spring project, Spring scheduling will be used to invoke the parser.

The following classes were created:

BlogRequest – invokes the parser on a given blog URL, passes parsed content to service layer
BlogRequestQueue – queues up and executes blog requests
BlogParser – interface with parseURL method
DishBlogParser – implements BlogParser, used to parse the blog The Dish

Each of these (aside from the interface) is configured as a Spring-managed bean. The code for BlogRequest:

@Component("blogRequest")
public class BlogRequest {
    @Autowired(required=true)
    private BlogParser parser;
    @Autowired(required=true)
    private BlogService service;
    @Autowired(required=true)
    private BlogRequestQueue requestQueue;
    /**
     * blogUrl URL to pages of the blog, in a format like
     * http://andrewsullivan.thedailybeast.com/page/##/ where
     * ## stands in for the page number.
     */
    @Value("${config.blogUrl}")
    private String blogUrl;

    /**
     * Execute a blog request for a given page number, invoking the parser,
     * and passing the data to the service layer for further processing.
     * @param pageNumber
     */
    public void execute(int pageNumber) {
        System.out.println("executing blog request");
        try {
            List links = parser.parseURL(new URL(blogUrl.replace("##", String.valueOf(pageNumber))));
            for (Link link : links) {
                service.addLinkAsync(link);
            }
            //if page > 1 and links not empty, queue a request for the next page
            if (pageNumber > 1 && !links.isEmpty())
                requestQueue.enqueue(pageNumber + 1);
        }

        catch (ParseException pe) {
            Logger.getLogger(BlogRequest.class.getName()).log(Level.SEVERE, null, pe);
            //reprocess
            requestQueue.enqueue(pageNumber);
        }
        catch (MalformedURLException mue) {
            Logger.getLogger(BlogRequest.class.getName()).log(Level.SEVERE, null, mue);
        }
    }
}

The @Component annotation allows Spring to autowire this bean to other beans. Here, we autowire in a parser, a service bean, and the request queue bean. blogUrl is configured in a properties file and @Value is used to wire it in.

The execute method invokes the parser, which returns a List of Link objects. A “Link”, in the context of this project, is a blog entry–specifically, its URL, posted date, excerpt, and assorted other data. This list is passed to the service bean for further processing. Provided certain conditions are met, the next page is added to the queue. In this way, older blog pages will be scraped one by one until the oldest is reached.

The code for BlogRequestQueue is pretty simple:

public class BlogRequestQueue {
    @Autowired(required=true)
    private BlogRequest blogRequest;
    private Queue queue = new LinkedList();
    /**
     * pageScanStart page number to initialize the queue to.
     * A value of zero or less will disable page scanning.
     */
    @Value("${config.pageScanStart}")
    private int pageScanStart = 0;

    @PostConstruct
    public void initializeQueue() {
        if (pageScanStart > 0)
            queue.add(pageScanStart);
    }

    public boolean enqueue(int pageNumber) {
        return queue.add(pageNumber);
    }

    /**
     * Execute the next request in the queue
     */
    //@Scheduled(fixedDelay=30000l)
    public synchronized void executeNext() {
        Integer pageNumber = queue.poll();
        if (pageNumber != null)
            blogRequest.execute(pageNumber);
    }

The value of pageScanStart is set via @Value from a properties file. The @PostConstruct annotation is used to mark initializeQueue() as an init method. Note that the @Scheduled annotation on the executeNext method is commented out. Rather than hard-code this setting, I moved the scheduling to applicationContext.xml to make the interval configurable.

BlogParser.java:

public interface BlogParser {

    public List parseURL(URL url) throws ParseException;

}

The code for DishBlogParser is where all the interesting web scraping happens. See my post web scraping Part 2 for the details.

A Blog Analyzer Project

James H. (admin) — Thu, 28 Apr 2011 17:51:35 +0000

In the coming days, I will be writing about a project I’m working on which will perform analysis on Andrew Sullivan’s The Dish blog, which is one of the most popular blogs on American politics. The intent of the project, which will utilize such technologies as Spring 3, JSP/JSTL, JDBC, PostgreSQL, and jQuery/Ajax, is to web scrape the blog, extract key data elements, and reorganize and present this data in new and interesting ways. Additionally, I will create a bookmarklet that will add value to the blog site itself.

Development tools used include Netbeans 7.0, Firefox with Firebug, and the always handy psql Postgres command line tool.

There are many interesting technical challenges involved, and I will write about them on this blog. Additionally, there is the question of copyright law, which is an unavoidable concern when building off of content from a third party. Copyright law was not meant to stifle innovation, though, provided certain criteria are met: the content originator must not be harmed in the marketplace, repurposed content must be transformed into a novel work, and small portions must be used. I believe my project fits these criteria.

Profile SQL statements in Java / Spring

James H. (admin) — Thu, 21 Apr 2011 19:34:12 +0000

Wouldn’t it be nice if there were a way to time your application’s SQL statements unobtrusively? This information could give you insight into the performance of your queries and updates and help you identify slow, poorly-performing SQL. Of course, there is a way to add such SQL profiling to your Spring application, by using AspectJ.

I use Spring JDBC and wanted to identify slow SQL queries in my application so that I could tune them in order to improve overall performance. Capturing the execution times whenever SQL is executed can be done by creating a pointcut on the methods of JdbcTemplate. Here is what we need:

aspectjrt.jar and aspectjweaver.jar from here.
An aspect with a pointcut on the JdbcOperations interface, which JdbcTemplate implements.
@Around advice that times the execution of JdbcTemplate methods and stores this data for later retrieval.
Configuration of Spring applicationContext.xml to get it working.

The pointcut looks like this, with the String argument being the SQL statement:

execution(* org.springframework.jdbc.core.JdbcOperations.*(String, ..))

Here is the profiling java class, with imports omitted (call it SqlProfiler):

@Aspect
public class SqlProfiler {
    private final Map sqlTimings;

    public SqlProfiler() {
        sqlTimings = Collections.synchronizedMap(new HashMap());
    }

    @Pointcut("execution(* org.springframework.jdbc.core.JdbcOperations.*(String, ..))")
    public void sql() {}

    @Around("sql()")
    public Object profile(ProceedingJoinPoint pjp) throws Throwable {
        long start = System.currentTimeMillis();
        Object obj = pjp.proceed();
        long time = System.currentTimeMillis() - start;
        String statement = pjp.getArgs()[0].toString();
        SqlTiming sqlTiming = null;
        synchronized(sqlTimings) {
            sqlTiming = sqlTimings.get(statement);
            if (sqlTiming == null) {
                sqlTiming = new SqlTiming(statement);
                sqlTimings.put(statement, sqlTiming);
            }
        }
        sqlTiming.recordTiming(time);

        return obj;
    }

    public List getTimings(final SortedBy sort) {
        List timings = new ArrayList(sqlTimings.values());
        Collections.sort(timings, new Comparator() {

            public int compare(SqlTiming o1, SqlTiming o2) {
                switch(sort) {
                    case AVG_EXECUTION_TIME:
                        return Math.round(o1.getAvgExecutionTime() - o2.getAvgExecutionTime());
                    case CUMULATIVE_EXECUTION_TIME:
                        long diff = o1.getCumulativeExecutionTime() - o2.getCumulativeExecutionTime();
                        if (diff > 0l)
                            return 1;
                        else if (diff == 0)
                            return 0;
                        else
                            return -1;
                    case NUMBER_OF_EXECUTIONS:
                        return o1.getExecutionCount() - o2.getExecutionCount();
                }
                return 0;
            }

        });

        return timings;
    }
}

To explain this a bit, sqlTimings is a map where keys are the SQL statements, and values are of type SqlTiming (shown below). The @Around advice applies to all methods that execute SQL. The profile method simply times the execution and stores the result in the map. The getTimings method is how clients can retrieve the profiling data that has been gathered, with several different sort options.

Here is the SqlTiming class, each instance of which holds the data for one SQL statement:

public class SqlTiming {

    private final String statement;
    private int count;
    private long cumulativeMillis;

    SqlTiming(String statement) {
        this.statement = statement;
    }

    synchronized SqlTiming recordTiming(long time) {
        count++;
        cumulativeMillis += time;
        return this;
    }

    public String getSqlStatement() {
        return statement;
    }

    public int getExecutionCount() {
        return count;
    }

    public long getCumulativeExecutionTime() {
        return cumulativeMillis;
    }

    public float getAvgExecutionTime() {
        return (float)cumulativeMillis / (float)count;
    }
}

Finally, we need to add the following to the Spring applicationContext.xml:

It’s that simple. To view the results that have been gathered (cumulative execution time, average execution time, and number of executions), call the getTimings method. Here I’ve outputted some of the data via JSP:

INSERT INTO link (link_id, url, link_date, title, exerpt, link_collection_id) VALUES (?, ?, ?, ?, ?, ?)
Avg execution: 122.26ms
Total # of executions: 34

SELECT link_id, url, link_date, title, exerpt FROM link WHERE url = ?
Avg execution: 123.39ms
Total # of executions: 690

UPDATE inner_link SET title = ?, refers_to_id = ? WHERE inner_link_id = ?
Avg execution: 124.91ms
Total # of executions: 526

And there you have it. To review: we’ve created an Aspect to profile SQL statements in Java unobtrusively, wired up our profiler in the Spring config, and outputted the SQL timings for analysis. Please leave a comment if this code can be improved!

Add custom annotation to Spring MVC controller

James H. (admin) — Mon, 04 Apr 2011 23:07:39 +0000

The question of how to add custom annotations to my Spring MVC controllers puzzled me for some time, because the documentation in this area is lacking. Even the bible of Spring 3.0, Spring Recipes by Gary Mak, et. al., did not address the topic.

But then I found this great blog post detailing exactly what I was interested in. In a nutshell, you need to implement WebArgumentResolver and set your class as a customArgumentResolver of the AnnotationMethodHandlerAdapter bean. What I was interested in was adding a @RequestAttribute annotation that would work like @RequestParam, but would obviously pull the value from a request attribute rather than a request parameter.

Following the tutorial in the link above, I created this annotation:

package my.package;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.PARAMETER)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface RequestAttribute {

    /**
     * The name of the request attribute to bind to.
     */
    String value() default "";

    /**
     * Whether the parameter is required.
     * Default is true, leading to an exception thrown in case
     * of the parameter missing in the request. Switch this to
     * false if you prefer a null in case of the parameter missing.
     * Alternatively, provide a {@link #defaultValue() defaultValue},
     * which implicitly sets this flag to false.
     */
    boolean required() default true;

    /**
     * The default value to use as a fallback. Supplying a default value
     * implicitly sets {@link #required()} to false.
     */
    String defaultValue() default "";
}

And the corresponding argument resolver:

package my.package;

import java.lang.annotation.Annotation;
import javax.servlet.http.HttpServletRequest;
import org.springframework.core.MethodParameter;
import org.springframework.web.bind.support.WebArgumentResolver;
import org.springframework.web.context.request.NativeWebRequest;

public class RequestAttributeArgumentResolver implements WebArgumentResolver {

    public Object resolveArgument(MethodParameter param,
                                  NativeWebRequest request) throws Exception {

        Annotation[] paramAnns = param.getParameterAnnotations();

        Class paramType = param.getParameterType();

        for (Annotation paramAnn : paramAnns) {
            if (RequestAttribute.class.isInstance(paramAnn)) {
                RequestAttribute reqAttr = (RequestAttribute) paramAnn;
                HttpServletRequest httprequest = (HttpServletRequest) request.getNativeRequest();
                Object result = httprequest.getAttribute(reqAttr.value());

                if (result == null)
                    result = reqAttr.defaultValue();

                if (result == null && reqAttr.required())
                    raiseMissingParameterException(reqAttr.value(), paramType);
                else
                    return result;
            }
        }

        return WebArgumentResolver.UNRESOLVED;
    }

    protected void raiseMissingParameterException(String paramName,
                                                  Class paramType) throws Exception {
        throw new IllegalStateException("Missing parameter '" + paramName
                                        + "' of type [" + paramType.getName() + "]");
    }
}

Just wire it up in your Spring configuration:

You may already be doing something with AnnotationMethodHandlerAdapter, such as setting a binding initializer, so in that case just add the customArgumentResolver to your existing config.