Skip to content

Posts from the ‘Java’ Category


Log SQL Statements With Parameter Values Filled In (Spring JDBC)

If you’re using parameterized queries with Spring JDBC (and why wouldn’t you?), it’s easy to log them with a simple log.debug() statement. But your logged statements will be full of ?’s instead of the values, which makes them much less useful. Suppose you’d like to know what was substituted for those question marks. It’s not so difficult if you use AspectJ.  Here is what is needed:

  1. aspectjrt.jar and aspectjweaver.jar from here.
  2. An aspect with a pointcut on the JdbcOperations interface, which JdbcTemplate implements.
  3. @Before advice that intercepts the execution of JdbcTemplate methods and logs de-parameterized SQL statements.
  4. Configuration of Spring applicationContext.xml to get it working.

Let’s start off with our class:

public class SqlLogger {
    private static final Logger log = LoggerFactory.getLogger(SqlLogger.class);

Here, I’m using an org.slf4j.Logger, but any logging framework will work.

Next step is to add the method that will capture the SQL and parameters as they are executed. Again, I’m using Spring JDBC, so all calls are made to an object that implements JdbcOperations.  We can set up our class to spy on all relevant calls by defining our method like so:

Read moreRead more


Configure Lucene IndexWriter and IndexSearcher in Spring applicationContext.xml

Problem: you want to define Lucene IndexWriter and IndexSearcher as beans inside your Spring application to be injected/autowired into other beans.

Solution: follow the following steps.

  • define the Lucene version as a constant
  • define a Lucene analyzer (StandardAnalyzer) as a bean
  • define a Lucene directory as a bean, using a factory-method for instantiation
  • define an IndexWriter, wiring in the Lucene directory and an IndexWriterConfig set to use your previously-defined analyzer
  • define an IndexSearcher, wiring in the Lucene directory
  • define also a query parser (StandardQueryParser), wiring in the analyzer bean

You can then wire/autowire these beans into your application beans, for example:
Read moreRead more


Simple pagination taglib for JSP

Pagination is a common requirement when writing JSPs. Long sets of data must be broken up across multiple pages. There is no standard way to implement pagination in JSP or JSTL, however. You must use a custom tag to generate the page numbers.

I’ve created a simple pagination taglib that generates the page links. You can customize it to your needs. Usage example below. Here, I’m using it to paginate search results from Lucene:

<c:url var="searchUri" value="/searchResults.html?s=${searchval}&page=##" />
<paginator:display maxLinks="10" currPage="${page}" totalPages="${totalPages}" uri="${searchUri}" />

The paginator:display tag produces this output:

shows links 1 through 10

Read moreRead more


Convert between Java enums and PostgreSQL enums

PostgreSQL allows you to create enum types using the following syntax:


You can now use ‘animal’ as a datatype in your tables, for example:

create table pet (                         
                  pet_id        integer         not null,
                  pet_type      animal_type     not null,
                  name          varchar(20)     not null

In Java, you’d have a corresponding enum type:

public enum AnimalType {

Converting between Java and PostgreSQL enums is straightforward. For example, to insert or update an enum field you could use the CAST syntax in your SQL PreparedStatement:

INSERT INTO pet (pet_id, pet_type, name) VALUES (?, CAST(? AS animal_type), ?);


INSERT INTO pet (pet_id, pet_type, name) VALUES (?, ?::animal_type, ?);

Postgres will also let you insert/update an enum just by passing its value as a string.

Whether casting or not, the Java side is the same. You would set the fields like this:

stmt.setInt(1, 1);
stmt.setString(2, AnimalType.DOG.toString());
stmt.setString(3, 'Rex');

Retrieving the enum from a SELECT statement looks like this:


Take into consideration that enums are case-sensitive, so any case mismatches between your Postgres enums and Java enums will have to be accounted for. Also note that the PostgreSQL enum type is non-standard SQL, and thus not portable.

Also, FYI, to view the set of values in a given Postgres enum type, you can use the following SQL query:

SELECT enumlabel FROM pg_enum 
    WHERE enumtypid = 'your_enum'::regtype ORDER BY oid;

Web scraping in Java with Jsoup, Part 2 (How-to)

Web scraping refers to programmatically downloading a page and traversing its DOM to extract the data you are interested in. I wrote a parser class in Java to perform the web scraping for my blog analyzer project. In Part 1 of this how-to I explained how I set up the calling mechanism for executing the parser against blog URLs. Here, I explain the parser class itself.

But before getting into the code, it is important to take note of the HTML structure of the document that will be parsed. The pages of The Dish are quite heavy–full of menus and javascript and other stuff, but the area of interest is the set of blog posts themselves. This example shows the HTML structure of each blog post on The Dish:
Read moreRead more


Disable URL session IDs (JSESSIONID) in Tomcat 7, Glassfish v3

URL-based session tracking is intended for web clients that do not support session cookies. Every browser worth mentioning supports these cookies, and almost nobody surfs with them disabled. Most web sites either state explicitly or assume that a user’s browser supports session cookies. URL rewriting schemes that add the session ID as a parameter on every URL thus provide very little benefit, if any at all. Session IDs showing up in URLs is just bad form, and may confuse search engine spiders. Thankfully the Servlet 3.0 standard gives you two ways to disable URL session rewriting. This works in Tomcat 7, Glassfish v3, and any other Servlet 3.0-compliant servlet container.

First, you can add this to your web.xml web-app config:


Or programmatically, you can use:


I’ve used the web.xml method in Tomcat 7, and it works. No jsessionid in the URLs when using <c:url …> in my JSPs.


Web scraping in Java with Jsoup, Part 1

In order to obtain the data to feed into my blog analyzer, content must be parsed from the pages of the blog itself.  This is called “web scraping”.  Jsoup will be used to parse the pages, and because this is a Spring project, Spring scheduling will be used to invoke the parser.

The following classes were created:

  • BlogRequest – invokes the parser on a given blog URL, passes parsed content to service layer
  • BlogRequestQueue – queues up and executes blog requests
  • BlogParser – interface with parseURL method
  • DishBlogParser – implements BlogParser, used to parse the blog The Dish

Each of these (aside from the interface) is configured as a Spring-managed bean.  The code for BlogRequest:
Read moreRead more


A Blog Analyzer Project

In the coming days, I will be writing about a project I’m working on which will perform analysis on Andrew Sullivan’s The Dish blog, which is one of the most popular blogs on American politics.  The intent of the project, which will utilize such technologies as Spring 3, JSP/JSTL, JDBC, PostgreSQL, and jQuery/Ajax, is to web scrape the blog, extract key data elements, and reorganize and present this data in new and interesting ways.  Additionally, I will create a bookmarklet that will add value to the blog site itself.

Development tools used include Netbeans 7.0, Firefox with Firebug, and the always handy psql Postgres command line tool.

There are many interesting technical challenges involved, and I will write about them on this blog.  Additionally, there is the question of copyright law, which is an unavoidable concern when building off of content from a third party.  Copyright law was not meant to stifle innovation, though, provided certain criteria are met: the content originator must not be harmed in the marketplace, repurposed content must be transformed into a novel work, and small portions must be used.  I believe my project fits these criteria.


Profile SQL statements in Java / Spring

Wouldn’t it be nice if there were a way to time your application’s SQL statements unobtrusively? This information could give you insight into the performance of your queries and updates and help you identify slow, poorly-performing SQL. Of course, there is a way to add such SQL profiling to your Spring application, by using AspectJ.

I use Spring JDBC and wanted to identify slow SQL queries in my application so that I could tune them in order to improve overall performance.  Capturing the execution times whenever SQL is executed can be done by creating a pointcut on the methods of JdbcTemplate.  Here is what we need:

  1. aspectjrt.jar and aspectjweaver.jar from here.
  2. An aspect with a pointcut on the JdbcOperations interface, which JdbcTemplate implements.
  3. @Around advice that times the execution of JdbcTemplate methods and stores this data for later retrieval.
  4. Configuration of Spring applicationContext.xml to get it working.

The pointcut looks like this, with the String argument being the SQL statement:
Read moreRead more


Add custom annotation to Spring MVC controller

The question of how to add custom annotations to my Spring MVC controllers puzzled me for some time, because the documentation in this area is lacking. Even the bible of Spring 3.0, Spring Recipes by Gary Mak, et. al., did not address the topic.

But then I found this great blog post detailing exactly what I was interested in.  In a nutshell, you need to implement WebArgumentResolver and set your class as a customArgumentResolver of the AnnotationMethodHandlerAdapter bean.  What I was interested in was adding a @RequestAttribute annotation that would work like @RequestParam, but would obviously pull the value from a request attribute rather than a request parameter.
Read moreRead more


JSON / Ajax in Spring MVC

How do you configure your Spring MVC web application to serve JSON for Ajax? It is not difficult. You probably have a view resolver in your dispatcher-servlet.xml that looks like this:

<bean id="viewResolver"

In order to serve JSON content, you need to use the ContentNegotiatingViewResolver. The configuration looks like this:
Read moreRead more


How to extract titles from web pages in Java

Let’s say you have a set of URLs and you want the web page titles associated with them. Maybe you’ve data-mined a bunch of links from HTML pages, or acquired a flat file listing URLs. How would you go about getting the corresponding page titles, and associating them with the URLs using Java?

You could use an HTML parser such as Jsoup to request the HTML document associated with each URL and parse it into a DOM document.  Once obtained, you could navigate the document and select the text from the title tag, like so:

String titleText ="title").first().text();

Elegant, but a lot of overhead for such a simple task. You’d be loading the whole page into memory and parsing it into a DOM structure just to extract the title.  Instead, you could use the Apache HTTP Client library, which provides a robust API for requesting resources over the HTTP protocol.  But it would be unnecessary in this case.  Let’s keep it simple and rely only on the java standard library.

Read moreRead more