By and large, the applications that we deploy are configured at runtime by using the Spring PropertyPlaceholderConfigurer. For those of you not familiar with this concept, basically, any environment specific properties in our application or even properties that we just want to be able to easily change, are defined using Ant style property syntax within our Spring beans configuration.

For instance, our DataSource configuration looks something like this:

<bean id="dataSource" class="com.atomikos.jdbc.nonxa.AtomikosNonXADataSourceBean"
          destroy-method="close">
      <property name="uniqueResourceName" value="My Primary Datasource"/>
      <property name="driverClassName" value="com.inet.tds.TdsDriver"/>
      <property name="url" value="jdbc:inetdae7a:${database.host}:${database.port}?database=${database.name}&amp;secureLevel=0"/>
      <property name="user" value="${database.user}"/>
      <property name="password" value="${database.password}"/>
      <property name="testQuery" value="select 1"/>
      <property name="maxPoolSize" value="50"/>
      <property name="minPoolSize" value="20"/>
      <!-- Setting reapTimeout=0 because atomikos incorrectly reaps nonxa connections. -->
      <property name="reapTimeout" value="0"></property>
    </bean>

By defining a PropertyPlaceholderConfigurer, we can have Spring inject the appropriate properties for this DataSource when the application starts up:

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>classpath:runtime-properties/${env}/my-application.properties</value>
        </property>
        <property name="nullValue">
            <value>null</value>
        </property>
        <!-- Force system properties to override any deployed runtime properties -->
        <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/>
    </bean>

For instance, my file runtime-properties/dev/my-application.properties would contain:

database.host=localhost
database.user=myuser
database.password=mypassword
database.name=myapplicationdb
database.port=1433

When we deploy our application, we deploy ALL of the runtime property files for all of the different runtime environments. By injecting the env property into the container at runtime, we tell the application which runtime properties to use, e.g.:

-Denv=dev

This is usually done either in the startup script or the startup configuration for the container. For example, in JBoss, we would add this to the run.conf file or the run.sh script.

This approach opens up some security issues, however. Now, all of my database credentials for all of my runtime environments are packaged up in cleartext in my single WAR artifact. If, for instance, the sys admin who manages our deployments in our testing environments (integration and staging) is not the same sys admin that manages our deployments in our production environment, especially if this is security related (only trusted senior administrators can manage production applications), we have a security hole because the credentials for all environments are stored in the WAR.

Historically, there has been no easy way around this. Typically, one would generate a secret key that could be used to decrypt the passwords if they were encrypted in the application configuration but then if you put this secret key IN the application deployment, you STILL have a security hole–any sufficiently determined adversary can use this key to decrypt your encrypted passwords.

Recently, I stumbled across the Jasypt project. It aims to provide some simplified interfaces into the Java Encryption API’s. The feature it offered that most appealed to me, however, was the integration that it offers for Spring that dovetails nicely with my configuration.

Jasypt offers a special implementation of the PropertyPlaceholderConfigurer called the EncryptablePropertyPlaceholderConfigurer that allows you to encrypt some of your properties in your configuration files and have them decrypted on the fly when the application is starting. What’s more important, is it allows you to pass the encryption key, or ‘password’ into the application as a system property at runtime, decoupling the knowledge of the key from the deployable artifact.

Let’s see how this would work

First we would need to decide on an encryption algorithm and a password. Then we would use the commandline tools that ship with Jasypt to encrypt our database password above:
./encrypt.sh input="mypassword" algorithm=PBEWithMD5AndTripleDES password=jasypt_is_cool
Jasypt will output something that looks like:

----ENVIRONMENT-----------------

Runtime: Sun Microsystems Inc. Java HotSpot(TM) Server VM 10.0-b23

—-ARGUMENTS——————-

algorithm: PBEWithMD5AndTripleDES
input: mypassword
password: jasypt_is_cool

—-OUTPUT———————-

AUP/WSfdvbAfVuBJW/kbXsqh2qu8yWiJ

Then, we update our configuration file and replace:

database.password=mypassword

with:

database.password=ENC(AUP/WSfdvbAfVuBJW/kbXsqh2qu8yWiJ)

The syntax ENC(...) tells the EncryptablePropertyPlaceholderConfigurer that this property is encrypted.

Now, we will add the beans for the encryptor and it’s configuration:

    <bean id="jasyptConfiguration"
          class="org.jasypt.encryption.pbe.config.EnvironmentStringPBEConfig"
          p:algorithm="PBEWithMD5AndTripleDES"
          p:passwordSysPropertyName="jasypt.encryption.password"/>

    <bean id="propertyPasswordEncryptor"
          class="org.jasypt.encryption.pbe.StandardPBEStringEncryptor"
          p:config-ref="jasyptConfiguration"/>

The important thing to take away from the above is the configuration. The property passwordSysPropertyName tells Jasypt that it should load the encryption password from a system property named jasypt.encryption.password.

Now, as I described above, I add a runtime property to my container so that this password gets injected into the runtime environment as a system property:

-Djasypt.encryption.password=jasypt_is_cool

The final step is to change the PropertyPlaceholderConfigurer and wire it up to this encryptor:

    <bean class="org.jasypt.spring.properties.EncryptablePropertyPlaceholderConfigurer">
        <constructor-arg>
          <ref bean="propertyPasswordEncryptor"/>
        </constructor-arg>
        <property name="locations">
          <value>classpath:runtime-properties/${env}/my-application.properties</value>
        </property>
        <property name="nullValue">
          <value>null</value>
        </property>
        <!-- Force system properties to override any deployed runtime properties -->
        <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/>
    </bean>

And that’s it! I’ve now added another layer of security to avoid having my production passwords floating around in my deployable artifact. It is generally a good idea to record this configuration in a secure location outside of source control that has limited access. This configuration should include the actual database password, the Jasypt encryption password used and the Jasypt algorithm that was used to generate the encrypted password, at a minimum.

Cheers!

Advertisements

Have you ever found yourself doing something like this in some code you’ve been writing?


String xml = "<adocument><anelement>" + someproperty + "</anelement><anotherelement>" 
                 + anotherproperty + "</anotherelement></adocument>";

Yuck!

I’m a lazy developer. And I know from experience that at some point, I’m going to have to change this embedded xml generating pile of crap. Someone will ask me to add an element, or remove an element, or add a namespace, or change the formatting, or …… Ugh! Maintaining this kind of code is what makes maintenance programmers go insane….

Fortunately, I’ve spent enough time doing maintenance development that I know that a little time spent upfront will save a lot of time and frustration down the road and I try to avoid naive solutions like the above in favor of something that appeals to my laziness.

One simple, yet effective, solution to the above problem is to use Velocity. Velocity is a powerful scripting language and makes short work of the simple example above. The main advantage is that the template is just text and text is easy to maintain.

Here’s how we’d go about refactoring the above solution…

First, I’d create a velocity template adocument.vm with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<adocument>
  <anelement>${someproperty}</anelement>
  <anotherelement>${anotherproperty}</anotherelement>
</adocument>

Then, I’d configure my velocity engine in my spring context:

    <bean id="velocityEngine" class="org.springframework.ui.velocity.VelocityEngineFactoryBean">
      <property name="velocityProperties">
       <value>
        resource.loader=class
        class.resource.loader.class=org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader
       </value>
      </property>
    </bean>

The properties tell Velocity how to find the template files when I give it the template file path. I’ve specified that I want Velocity to use the ClasspathResourceLoader to find my templates, which are likely packaged up inside my application archive (war, jar).

Now, I just wire up the class that is generating the xml to the engine:

private VelocityEngine engine;

public void setVelocityEngine(VelocityEngine engine)
{
    this.engine = engine;
}
<property name="velocityEngine" ref="velocityEngine"/>

And tell it where the template is:

String templateFile;

public void setTemplateFile(String file)
{
    this.templateFile = file;
}
<property name="templateFile" value="myTemplatesDirectory/adocument.vm"/>

myTemplatesDirectory should be a directory in your classpath (packaged in your jar?) that contains your template.

Then, I change how I’m generating the xml:

Map model = new HashMap();
model.put("someproperty", someproperty);
model.put("anotherproperty", anotherproperty);
String xml = VelocityEngineUtils.mergeTemplateIntoString(velocityEngine, templateFile, model);

Note that this solution still violates the open closed principle but that can be solved by making the generation code a reuseable module and having the client pass in the variables:

public String generateContent(Map model, String template)
{
    return VelocityEngineUtils.mergeTemplateIntoString(velocityEngine, template, model);
}

You may still need to update how the model is generated if the view needs to change (e.g. to add a new field) but you won’t have to muck around with any of the software to make simple textual changes to the xml. That’s a solution that appeals to my laziness.

It should be obvious to some of you that this refactoring is essentially a simple MVC solution. Velocity can be, and frequently is, used in place of solutions like jsp to generate views. However, as shown above, it is a powerful tool in your toolbox that can easily be used in your middleware for generating content that could, for instance, become the payload of a JMS message.

Cheers!

I’m currently working on an application that allows an administrative user to upload an input file that contains thousands of records that need to be processed.

Our initial naive solution to this problem was to parse the upload file into individual records, save the record to the database and then send a JMS message containing the PK of the record to a JMS queue to kick off asynchronous processing for each record.

It worked great….until we tried to process a file with more than a few thousand records. Then the transaction that the file processing was running in started timing out. That’s a problem.

You see, our benchmark for file processing is 10k records–that’s the expected size of the files we will start receiving once our system goes live. We’d also like to have some additional breathing room in processing capability in case we see larger files in the wild. But, as it was, not only did we not have any breathing room, we had a giant boulder sitting on our chest crushing us.

What to do?

It was about this time that I reflected on the need for an entirely new approach, not a variation on our existing processing strategy. Spring Batch to the rescue.

The Spring Batch framework is a new addition to the Spring portfolio. The primary committers on the project have extensive experience producing batch processing solutions. It is primarily designed as a Java based batch processing framework using traditional batch processing strategies. In other words, it wasn’t a natural fit for my needs: kicking off batch processing with dynamic input from within a web application.

Many (most) of the collaborators in a spring batch configuration are stateful so they don’t lend themselves to traditional singleton based spring context wiring. I originally tried defining these beans as prototypes but the problem was that one of the stateful collaborators needs to be injected TWICE into one of the components (as a collaborator and to register as a listener). It couldn’t be defined as a prototype or two different objects would be injected and that wouldn’t work.

In the documentation and on the forums, the Spring Batch team has suggested that you just use a new ApplicationContext for each run of your batch job. All of the examples that I’ve found construct the ENTIRE application context for each run of the batch job(s). That won’t work in my case because my application context is being used by the entire web application. I can’t refresh it (and the hundreds of beans it contains) just to breath new life into a handful of stateful spring batch beans.

Enter the ClassPathXmlApplicationContextJobFactory. This class allows you to construct your batch job beans from an existing parent ApplicationContext and a subcontext bean definition file. The key to this is that every time you request a job from the factory, it constructs a brand new sub context and wires up your batch job beans to the rest of it’s singleton collaborators in your parent application context. Yippee!

The only gripe I had about this JobFactory implementation was that it expected the properties to be injected as constructor arguments (including the parent application context) and I wasn’t aware of a way to pass the application context into a bean unless the bean implemented ApplicationContextAware. Hence, I created the following wrapper class that supports property injection making configuration a snap:

public class ContextAwareJobFactory implements JobFactory, ApplicationContextAware, InitializingBean
{
    private ClassPathXmlApplicationContextJobFactory delegate;
    
    /* The parent application context */
    private ApplicationContext applicationContext;
    /* The job bean name */
    private String beanName;
    /* resource path to subcontext spring config */
    private String subcontextPath;

    public void setBeanName(String beanName)
    {
        this.beanName = beanName;
    }

    public void setSubcontextPath(String subcontextPath)
    {
        this.subcontextPath = subcontextPath;
    }

    /* (non-Javadoc)
     * @see org.springframework.batch.core.configuration.JobFactory#createJob()
     */
    @Override
    public Job createJob()
    {
        return delegate.createJob();
    }

    /* (non-Javadoc)
     * @see org.springframework.batch.core.configuration.JobFactory#getJobName()
     */
    @Override
    public String getJobName()
    {
        return delegate.getJobName();
    }

    /* (non-Javadoc)
     * @see org.springframework.context.ApplicationContextAware#setApplicationContext(org.springframework.context.ApplicationContext)
     */
    @Override
    public void setApplicationContext(ApplicationContext context) throws BeansException
    {
        this.applicationContext = context;
    }

    /* (non-Javadoc)
     * @see org.springframework.beans.factory.InitializingBean#afterPropertiesSet()
     */
    @Override
    public void afterPropertiesSet() throws Exception
    {
        delegate = new ClassPathXmlApplicationContextJobFactory(this.beanName, this.subcontextPath, this.applicationContext);
    }
}

Configuration is then very simple:

    <bean id="jobFactory" class="com.mybatch.ContextAwareJobFactory">
      <property name="beanName" value="myJob"/>
      <property name="subcontextPath" value="classpath:spring/mybatch-processing-prototype-beans.xml"/>
    </bean>

Note that all of the stateless beans that my batch process uses (e.g. FieldSetMappers, LineTokenizers, etc) are stored in the parent context. It’s only beans that are either stateful or are injected with stateful beans that are defined in the sub context “prototype” beans file.

And that’s it! Presto, I’m able to process input files with thousands or tens of thousands of records with no problem. Spring Batch also supports chunking and restart so if your batch job gets interrupted, it can be restarted again and pickup where it left off and continue processing.

If you are Spring addicted and find yourself in need of a batch processing solution, I’d suggest that you give Spring Batch a long look.