Java Ninja Chronicles By Norris Shelton

Things I learned in the pursuit of code

I had come up with a Springframework Junit configuration for Spring Batch that worked pretty well. here. This worked pretty well, but I wanted to have the ability to have transactions to rollback my test data for test repeatability. After much tinkering, this is what I have come up with.

Spring Batch Test Maven Dependencies

I added the following dependencies to write the Spring Batch tests.

<dependency>
    <groupId>org.springframework.batch</groupId>
    <artifactId>spring-batch-test</artifactId>
    <version>${spring.batch.version}</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-test</artifactId>
    <version>${spring.version}</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>

Spring Batch Test Utilities

Spring Batch provides the JobLauncherTestUtils to make it easier to test jobs. The gist of a job test class is:

JobExecution jobExecution = jobLauncherTestUtils.launchJob();

To test a step, you provide the step name to the lanuchStep method.

JobExecution jobExecution = jobLauncherTestUtils.launchStep("step1")

My spring test context defined the JobLauncherTestUtils.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <import resource="classpath:applicationContext.xml"/>

    <bean id="jobLauncherTestUtils" class="org.springframework.batch.test.JobLauncherTestUtils"/>
</beans>

Job and Step Test

My Spring Junit test class is fairly simple. This is no different than any other Job or Step test class. NOTE: I was not able to get transactions to work for a Job or a Step.

package com.javaninja.batch;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.test.JobLauncherTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import java.util.List;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;

/**
 * @author norris.shelton
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration
public class TestJobAndStep {

    @Autowired
    private JobLauncherTestUtils jobLauncherTestUtils;

    @Test
    public void testJob() throws Exception {
        commonAssertions(jobLauncherTestUtils.launchJob());
    }

    @Test
    public void testStep1() throws Exception {
        commonAssertions(jobLauncherTestUtils.launchStep("step1"));
    }

    private void commonAssertions(JobExecution jobExecution) {
        assertNotNull(jobExecution);

        BatchStatus batchStatus = jobExecution.getStatus();
        assertEquals(BatchStatus.COMPLETED, batchStatus);
        assertFalse(batchStatus.isUnsuccessful());

        ExitStatus exitStatus = jobExecution.getExitStatus();
        assertEquals("COMPLETED", exitStatus.getExitCode());
        assertEquals("", exitStatus.getExitDescription());

        List<Throwable> failureExceptions = jobExecution.getFailureExceptions();
        assertNotNull(failureExceptions);
        assertTrue(failureExceptions.isEmpty());
    }
}

Testing Step-scope components

I was able to have more success with transactions when testing Step-scope components like the JPA-related reader and writer. By using the StepScopeTestExecutionListener in combination with TransactionalTestExecutionListener, I was able to get transactions to work correctly.

The JpaPagingItemReader has the following methods that you need to be concerned with:

  • open – opens the output source.
  • read – reads the data.
  • close – closes the entity manager.

The JpaItemWriter provides the write method that handles all of the writing duties, including flushing the data.

The reader and writer test class is:

package com.javaninja.batch;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.item.ItemStreamException;
import org.springframework.batch.item.database.JpaItemWriter;
import org.springframework.batch.item.database.JpaPagingItemReader;
import org.springframework.batch.test.MetaDataInstanceFactory;
import org.springframework.batch.test.StepScopeTestExecutionListener;
import org.springframework.batch.test.StepScopeTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.TestExecutionListeners;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;
import org.springframework.test.context.transaction.TransactionalTestExecutionListener;
import org.springframework.transaction.annotation.Transactional;

import java.util.LinkedList;
import java.util.List;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.fail;

/**
 * @author norris.shelton
 */
@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners({DependencyInjectionTestExecutionListener.class,
                         StepScopeTestExecutionListener.class,
                         TransactionalTestExecutionListener.class})
@Transactional
@ContextConfiguration
public class TestReaderAndWriter {

    @Autowired
    private JpaPagingItemReader<CamAffiliateEntity> itemReader;

    @Autowired
    private JpaItemWriter<CamAffiliateEntity> itemWriter;

    @Test
    public void testReader() {
        StepExecution execution = MetaDataInstanceFactory.createStepExecution();
        int count = 0;
        try {
            count = StepScopeTestUtils.doInStepScope(execution, () -> {
                int numStates = 0;
                itemReader.open(execution.getExecutionContext());
                CamAffiliateEntity camAffiliateEntity;
                try {
                    while ((camAffiliateEntity = itemReader.read()) != null) {
                        assertNotNull(camAffiliateEntity);
                        assertNotNull(camAffiliateEntity.getAffiliateId());
                        assertNotNull(camAffiliateEntity.getName());
                        assertNotNull(camAffiliateEntity.getChannelId());
                        numStates++;
                    }
                } finally {
                    try { itemReader.close(); } catch (ItemStreamException e) { fail(e.toString());
                    }
                }
                return numStates;
            });
        } catch (Exception e) {
            fail(e.toString());
        }
        assertEquals(12, count);
    }

    @Test
    public void testWriter() throws Exception {
        List<CamAffiliateEntity> usStateEntities = new LinkedList<>();
        CamAffiliateEntity usStateEntity;
        for (int i = 0; i < 100; i++) {
            usStateEntity = new CamAffiliateEntity();
            usStateEntity.setAffiliateId(i);
            usStateEntity.setName("TEST-DELETE-" + i);
            usStateEntity.setChannelId(13);  // test
            usStateEntities.add(usStateEntity);
        }

        StepExecution execution = MetaDataInstanceFactory.createStepExecution();
        StepScopeTestUtils.doInStepScope(execution, () -> {
            itemWriter.write(usStateEntities);
            return null;
        });
    }
}

Summary

This summary provided me with the ability to run by Spring Batch JPA project for the reader and writer repeatedly without having test data-related problems.

The entire project used to write this blog is located on GitHub sheltonn / spring-batch-jpa

February 19th, 2016

Posted In: hibernate, Integration Tests, Java, java ninja, Javaninja, JUnit, Spring, Spring Batch, Test Driven Development, Unit Tests

Leave a Comment

This is a simple Spring Batch project. This implementation will read from a database table and write to a database table via JPA.

Maven Dependencies

The Maven dependency for Spring Batch is:

<dependency>
    <groupId>org.springframework.batch</groupId>
    <artifactId>spring-batch-core</artifactId>
    <version>${spring.batch.version}</version>
</dependency>

Please note that Spring Batch does use the Spring Framework. Spring Batch 3.0.6 corresponds to Spring 4.0.5. Attempting to use a newer version of Spring will cause runtime errors.

You will also need dependencies for communicating with the database.

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-orm</artifactId>
    <version>${spring.version}</version>
</dependency>
<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-entitymanager</artifactId>
    <version>5.0.7.Final</version>
</dependency>
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-dbcp2</artifactId>
    <version>2.1</version>
</dependency>
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.38</version>
</dependency>

Below is my entire pom.xml.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.javaninja</groupId>
    <artifactId>spring-batch-jpa</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <!--This is the version of Spring that Spring Batch uses-->
        <spring.version>4.0.5.RELEASE</spring.version>
        <spring.batch.version>3.0.6.RELEASE</spring.batch.version>
        <slf4j.version>1.7.13</slf4j.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-core</artifactId>
            <version>${spring.batch.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-orm</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-entitymanager</artifactId>
            <version>5.0.7.Final</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-dbcp2</artifactId>
            <version>2.1</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.38</version>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.4</version>
        </dependency>

        <!--
            Logging
        -->
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.1.3</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>log4j-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>

        <!--
            Testing
        -->
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-test</artifactId>
            <version>${spring.batch.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Spring and Spring Batch Context

Job Configuration

You will need the standard JobRepository, JobLauncher and TransactionManager (See Database/JPA below).

<!--This repository is only really intended for use in testing and rapid prototyping-->
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"/>
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"
      p:jobRepository-ref="jobRepository"/>

JpaPagingItemReader

The reading of data from the database via JPA is handled by a JpaPagingItemReader. The JpaPagingItemReader requires the following:

  • EntityManagerFactory – manages the entities (see JPA configuration below).
  • queryString – defines the query used to read the entities. In this case, it reads all entities of a given type.

The configuration for the JpaPagingItemReader is:

<!-- ItemReader which reads data from the database -->
<bean id="itemReader" class="org.springframework.batch.item.database.JpaPagingItemReader"
      p:entityManagerFactory-ref="entityManagerFactory"
      p:queryString="SELECT s FROM CamAffiliateEntity s"/>

JpaItemWriter

The writing of the JPA data is handled by JpaItemWriter. The JpaItemWriter also requires an EntityManagerFactory (see JPA configuration below).

The configuration for the JpaItemWriter is:

<!-- ItemWriter which writes the data to the database -->
<bean id="itemWriter" class="org.springframework.batch.item.database.JpaItemWriter"
    p:entityManagerFactory-ref="entityManagerFactory"/>

JPA Configuration

You will also need the following to configure the Jpa-support to read the records from the database:

  • EntityManagerFactory – manages the entities.
  • JpaVendorAdapter – exposes vendor-specific JPA properties.
  • DataSource – manages database connections.
  • TransactionManager – manages transactions.

Spring Batch requires a transaction manager and so does Spring’s JPA support. In this case, I’m using the same transaction manager for both. I’m not sure if this is correct, but it appears to work.

<bean id="dataSource" class="org.apache.commons.dbcp2.BasicDataSource" destroy-method="close"
      p:driverClassName="com.mysql.jdbc.Driver"
      p:url="jdbc:mysql://database.javaninja.com/batch?zeroDateTimeBehavior=convertToNull"
      p:username="javaninja"
      p:password="javaninja"/>
<!-- JPA EntityManagerFactory configuration -->
<bean id="jpaVendorAdapter" class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"
      p:dataSource-ref="dataSource"
      p:packagesToScan="com.javaninja.batch"
      p:jpaVendorAdapter-ref="jpaVendorAdapter"/>
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager"/>

The entire Spring context is below:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans   http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <import resource="classpath:META-INF/spring/batchContext.xml"/>

    <context:component-scan base-package="com.javaninja.batch"/>


    <!--This repository is only really intended for use in testing and rapid prototyping-->
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"/>

    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"
          p:jobRepository-ref="jobRepository"/>



    <bean id="dataSource" class="org.apache.commons.dbcp2.BasicDataSource" destroy-method="close"
          p:driverClassName="com.mysql.jdbc.Driver"
          p:url="jdbc:mysql://10.20.13.53/cam?zeroDateTimeBehavior=convertToNull"
          p:username="usr_ig_dev"
          p:password="igaming_developer"/>

    <!-- JPA EntityManagerFactory configuration -->
    <bean id="jpaVendorAdapter" class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>

    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"
          p:dataSource-ref="dataSource"
          p:packagesToScan="com.javaninja.batch"
          p:jpaVendorAdapter-ref="jpaVendorAdapter"/>

    <bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager"/>

    <!-- ItemReader which reads data from the database -->
    <bean id="itemReader" class="org.springframework.batch.item.database.JpaPagingItemReader"
          p:entityManagerFactory-ref="entityManagerFactory"
          p:queryString="SELECT s FROM CamAffiliateEntity s"/>

    <!-- ItemWriter which writes the data to the database -->
    <bean id="itemWriter" class="org.springframework.batch.item.database.JpaItemWriter"
        p:entityManagerFactory-ref="entityManagerFactory"/>

</beans>

Job Configuration

The XML configuration for the JPA batch job is pretty much the same as my other examples.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:batch="http://www.springframework.org/schema/batch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">

    <batch:job id="carJob">
        <batch:step id="step1">
            <batch:tasklet>
                <batch:chunk reader="itemReader" writer="itemWriter" commit-interval="10"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>

</beans>

The entire project used to write this blog is located on GitHub sheltonn / spring-batch-jpa

February 19th, 2016

Posted In: Java, java ninja, Javaninja, Spring, Spring Batch

Leave a Comment

Spring-Batch provides many types of readers and writers. The previous article, Spring-Batch – Reading and Writing XML provided the configuration needed to read and write XML. In this installment, I will present the configuration needed to read and write CSV files.

Maven Dependencies

You will need the Spring-Batch dependency.

<dependency>
    <groupId>org.springframework.batch</groupId>
    <artifactId>spring-batch-core</artifactId>
    <version>${spring.batch.version}</version>
</dependency>

You do not need a library for the CSV functionality.

The entire pom.xml looks like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.javaninja</groupId>
    <artifactId>spring-batch-csv</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <!--This is the version of Spring that Spring Batch uses-->
        <spring.version>4.0.5.RELEASE</spring.version>
        <spring.batch.version>3.0.6.RELEASE</spring.batch.version>
        <slf4j.version>1.7.13</slf4j.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-core</artifactId>
            <version>${spring.batch.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!--
            Logging
        -->
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.1.3</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>log4j-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>

        <!--
            Testing
        -->
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-test</artifactId>
            <version>${spring.batch.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Spring and Spring-Batch Context

Job Configuration

You will need the standard JobRepository, JobLauncher and TransactionManager.

<bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
<!--This repository is only really intended for use in testing and rapid prototyping-->
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"
      p:transactionManager-ref="transactionManager"/>
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"
      p:jobRepository-ref="jobRepository"/>

FlatFileItemReader

The reading of a CSV is handled by a FlatFileItemReader. You will also need the following to configure the ItemReader to read the lines of CSV data into objects:

  • LineMapper – Maps lines in a flat file to objects.
  • LineTokenizer – Splits lines of text by a delimiter.
  • FieldSetMapper – Maps fields into an object.

The Spring XML configuration for the ItemReader looks like:

<bean id="fieldSetMapper" class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper"
      p:targetType="com.javaninja.batch.Car"/>
<bean id="delimitedLineTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer
      p:names="make, model, color, doors"/>
<bean id="lineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper"
      p:lineTokenizer-ref="delimitedLineTokenizer"
      p:fieldSetMapper-ref="fieldSetMapper"/>
<!-- ItemReader which reads data from CSV file -->
<bean id="csvItemReader" class="org.springframework.batch.item.file.FlatFileItemReader"
      p:resource="classpath:cars-input.xml"
      p:lineMapper-ref="lineMapper"/>

FlatFileItemWriter

The writing of the CSV file is handled by a FlatFileItemWriter. You will also need the following to configure the item writer to write the objects as lines of CSV data:

  • LineAggregator – Maps the object into a string representation.
  • FieldExtractor – Extracts the specified properties of a bean in the specified order.

The Spring XML configuration for the ItemWriter looks like:

<bean id="fieldExtractor" class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor"
      p:names="make, model, color, doors"/>
<bean id="lineAggregator" class="org.springframework.batch.item.file.transform.DelimitedLineAggregator"
      p:fieldExtractor-ref="fieldExtractor"/>
<!-- ItemWriter which writes the data in CSV format -->
<bean id="csvItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"
      p:resource="file:csv/cars.xml"
      p:lineAggregator-ref="lineAggregator"/>

The entire Spring context looks like:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans   http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <import resource="classpath:META-INF/spring/batchContext.xml"/>

    <context:component-scan base-package="com.javaninja.batch"/>

    <bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>

    <!--This repository is only really intended for use in testing and rapid prototyping-->
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"
          p:transactionManager-ref="transactionManager"/>

    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"
          p:jobRepository-ref="jobRepository"/>



    <bean id="fieldSetMapper" class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper"
          p:targetType="com.javaninja.batch.Car"/>

    <bean id="delimitedLineTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer"
          p:names="make, model, color, doors"/>

    <bean id="lineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper"
          p:lineTokenizer-ref="delimitedLineTokenizer"
          p:fieldSetMapper-ref="fieldSetMapper"/>

    <!-- ItemReader which reads data from CSV file -->
    <bean id="csvItemReader" class="org.springframework.batch.item.file.FlatFileItemReader"
          p:resource="classpath:cars-input.xml"
          p:lineMapper-ref="lineMapper"/>



    <bean id="fieldExtractor" class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor"
          p:names="make, model, color, doors"/>

    <bean id="lineAggregator" class="org.springframework.batch.item.file.transform.DelimitedLineAggregator"
          p:fieldExtractor-ref="fieldExtractor"/>

    <!-- ItemWriter which writes the data in CSV format -->
    <bean id="csvItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"
          p:resource="file:csv/cars.xml"
          p:lineAggregator-ref="lineAggregator"/>

</beans>

Job configuration

The XML configuration for the CSV batch job is pretty much the same as any other example.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:batch="http://www.springframework.org/schema/batch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">

    <batch:job id="carJob">
        <batch:step id="step1">
            <batch:tasklet>
                <batch:chunk reader="csvItemReader" writer="csvItemWriter" commit-interval="1000"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>

</beans>

Running a job is the same way as in the XML example. Spring-Batch – Reading and Writing XML

The entire project used to write this blog is located on GitHub sheltonn / spring-batch-csv

February 18th, 2016

Posted In: Java, java ninja, Javaninja, Spring Batch

Leave a Comment

Once you have a Spring Batch application written, how do you test it?

This is a follow-on of Spring-Batch – Reading and Writing XML

Testing a Spring Batch Job or a Step

I used the following dependencies to write the Spring Batch tests.

<dependency>
    <groupId>org.springframework.batch</groupId>
    <artifactId>spring-batch-test</artifactId>
    <version>${spring.batch.version}</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-test</artifactId>
    <version>${spring.version}</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>

Spring Batch provides JobLauncherTestUtils to make it easier to test jobs. The gist of a job test class is:

JobExecution jobExecution = jobLauncherTestUtils.launchJob();

To test a step, you provide the step name to the launchStep method.

JobExecution jobExecution = jobLauncherTestUtils.launchStep("step1")

My Spring test context defined the JobLauncherTestUtils

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <import resource="classpath:applicationContext.xml"/>

    <bean id="jobLauncherTestUtils" class="org.springframework.batch.test.JobLauncherTestUtils"/>
</beans>

My Spring Junit test class is fairly simple.

package com.javaninja.batch;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.test.JobLauncherTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import java.util.List;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;

/**
 * @author norris.shelton
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration
public class TestJobAndStep {

    @Autowired
    private JobLauncherTestUtils jobLauncherTestUtils;

    @Test
    public void testJob() throws Exception {
        commonAssertions(jobLauncherTestUtils.launchJob());
    }

    @Test
    public void testStep1() throws Exception {
        commonAssertions(jobLauncherTestUtils.launchStep("step1"));
    }

    private void commonAssertions(JobExecution jobExecution) {
        assertNotNull(jobExecution);

        BatchStatus batchStatus = jobExecution.getStatus();
        assertEquals(BatchStatus.COMPLETED, batchStatus);
        assertFalse(batchStatus.isUnsuccessful());

        ExitStatus exitStatus = jobExecution.getExitStatus();
        assertEquals("COMPLETED", exitStatus.getExitCode());
        assertEquals("", exitStatus.getExitDescription());

        List<Throwable> failureExceptions = jobExecution.getFailureExceptions();
        assertNotNull(failureExceptions);
        assertTrue(failureExceptions.isEmpty());
    }
}

Testing Spring Batch Step-scope objects (readers and writers)

Spring Batch provides a StepScopeTestExecutionListener to allow it to inject Step-scope items into your test class via the normal @Autowired.

Readers and writers have three methods that you need to be concerned with.

  • open – Opens the output source.
  • read or write – reads or writes the data, respectively.
  • close – Flushes and closes the output source.

Spring Batch provides a MetaDataInstanceFactory to create a step execution with default parameters.

Spring Batch also provides a StepScopeTestUtils to assist in testing steps with the objects that they would have in scope during their step. To write the test, you will need to implement the doInStepScope method. This method requires a Callable to be implemented. I implemented mine as a Lambda-expression.

Please note that when you call the read method, it will read one item. When you call the write method, it will write the entire collection of provided data.

This all comes together in my test class.

package com.javaninja.batch;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.item.ItemStreamException;
import org.springframework.batch.item.xml.StaxEventItemReader;
import org.springframework.batch.item.xml.StaxEventItemWriter;
import org.springframework.batch.test.MetaDataInstanceFactory;
import org.springframework.batch.test.StepScopeTestExecutionListener;
import org.springframework.batch.test.StepScopeTestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.TestExecutionListeners;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;

import java.util.LinkedList;
import java.util.List;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;

/**
 * @author norris.shelton
 */
@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners({DependencyInjectionTestExecutionListener.class, StepScopeTestExecutionListener.class})
@ContextConfiguration
public class TestReaderAndWriter {

    @Autowired
    private StaxEventItemReader<Car> itemReader;

    @Autowired
    private StaxEventItemWriter<Car> itemWriter;

    @Test
    public void testReader() {
        StepExecution execution = MetaDataInstanceFactory.createStepExecution();
        int count = 0;
        try {
            count = StepScopeTestUtils.doInStepScope(execution, () -> {
                int numCars = 0;
                itemReader.open(execution.getExecutionContext());
                Car car;
                try {
                    while ((car = itemReader.read()) != null) {
                        assertNotNull(car);
                        assertNotNull(car.getMake());
                        assertNotNull(car.getModel());
                        assertNotNull(car.getColor());
                        assertTrue(car.getDoors() > 0);
                        numCars++;
                    }
                } finally {
                    try { itemReader.close(); } catch (ItemStreamException e) { fail(e.toString());
                    }
                }
                return numCars;
            });
        } catch (Exception e) {
            fail(e.toString());
        }
        assertEquals(100000, count);
    }

    @Test
    public void testWriter() throws Exception {
        List<Car> cars = new LinkedList<>();
        Car car;
        for (int i = 1; i < 10001; i++) {
            car = new Car();
            car.setMake("make" + i);
            car.setModel("model" + i);
            car.setColor("color" + i);
            car.setDoors(i);
            cars.add(car);
        }

        StepExecution execution = MetaDataInstanceFactory.createStepExecution();
        StepScopeTestUtils.doInStepScope(execution, () -> {
            itemWriter.open(execution.getExecutionContext());
            itemWriter.write(cars);
            itemWriter.close();
            return null;
        });
    }
}

February 18th, 2016

Posted In: Java, java ninja, Javaninja, JUnit, Spring Batch, Unit Tests

2 Comments

In Spring-Batch – Reading and Writing XML there was a note on the JobRepository in the src/main/resources/applicationContext.xml. The note clearly states that the Class that was used isn’t ready for prime time. The problem is that this job repository is stored in memory only. This is a problem if a job has to be restarted. As a reference, here is the snippet around the JobRepository

    <!--This repository is only really intended for use in testing and rapid prototyping-->
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"
          p:transactionManager-ref="transactionManager"/>

If that isn’t the real-deal, then what is?

Spring-Batch JobRepositoryFactoryBean

The MapJobRepositoryFactoryBean extends AbstractJobRepositoryFactoryBean. The only other class that extends it is JobRepositoryFactoryBean. This uses a database to store the job related information. This is how it is declared.

<bean id="jobRepository"
    class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
    <property name="dataSource" ref="dataSource" />
    <property name="transactionManager" ref="transactionManager" />
    <property name="databaseType" value="mysql" />
</bean>
  • dataSource is a normal database connection.
  • transactionManager is also a standard database transaction manager.
  • databaseType indicates the type of database. See org.springframework.batch.support.DatabaseType to determine the correct type. The values can be entered in lower-case.

Spring Batch Schema tables

Another thing you will need to do is add the database-related tables to store the information.

<jdbc:initialize-database data-source="dataSource">
    <jdbc:script location="org/springframework/batch/core/schema-drop-mysql.sql" />
    <jdbc:script location="org/springframework/batch/core/schema-mysql.sql" />
  </jdbc:initialize-database>

The scripts follow the pattern of schema-drop-{databaseType}.sql and schema-{databaseType}.sql. There are scripts for lots and lots of different databases. The scripts are located in spring-batch-core jar in org/springframework/batch/core.

The following database types are supported and have scripts:

  • db2 – IBM DB2
  • derby – Apache Derby
  • h2 – H2
  • hsqldb – HSQL Database Engine
  • mysql – MySQL
  • oracle – Oracle (10g)
  • postgresql – PostgreSQL
  • sqlf – Pivotal SQLFire
  • sqlite – SQLite
  • sqlserver – Microsoft SQL Server
  • sybase – Sybase

There is your answer.

February 18th, 2016

Posted In: Java, java ninja, Javaninja, Spring Batch

One Comment

This is a simple Spring Batch project. This implementation will read an XML file and write an XML file with almost no Java code.

Maven Dependencies

The Maven dependency for Spring Batch is:

<dependency>
    <groupId>org.springframework.batch</groupId>
    <artifactId>spring-batch-core</artifactId>
    <version>${spring.batch.version}</version>
</dependency>

Please note that Spring Batch does use the Spring Framework. Spring Batch 3.0.6 corresponds to Spring 4.0.5. Attempting to use a newer version of Spring will cause runtime errors.

You will also need to include the spring-oxm dependency to pull in the XML to object mapping library.

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-oxm</artifactId>
    <version>${spring.version}</version>
    <exclusions>
        <exclusion>
            <artifactId>commons-logging</artifactId>
            <groupId>commons-logging</groupId>
        </exclusion>
    </exclusions>
</dependency>

Below is my entire pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.javaninja</groupId>
    <artifactId>spring-batch-xml</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <!--This is the version of Spring that Spring Batch uses-->
        <spring.version>4.0.5.RELEASE</spring.version>
        <spring.batch.version>3.0.6.RELEASE</spring.batch.version>
        <slf4j.version>1.7.13</slf4j.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-core</artifactId>
            <version>${spring.batch.version}</version>
        </dependency>
        <!-- Spring XML processing-->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-oxm</artifactId>
            <version>${spring.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>commons-logging</artifactId>
                    <groupId>commons-logging</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <!--
            Logging
        -->
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.1.3</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>log4j-over-slf4j</artifactId>
            <version>${slf4j.version}</version>
        </dependency>

        <!--
            Testing
        -->
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-test</artifactId>
            <version>${spring.batch.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Spring and Spring Batch Context

Spring Batch has a few components that will be common across all of your jobs.

  • JobRepository – This is a repository that holds jobs.
  • JobLauncher – This is used to launch a job.
  • TransactionManager – This is a Batch TransactionManager, not to be confused with the Transaction Managers that you are used to dealing with.

Spring Batch uses Readers and Writers to read and write data. I am going to read and write XML data. To accomplish this, I will use a StaxEventItemReader and a StaxEventItemWriter. As part of the Stax readers and writers, I defined a Jaxb2Marshaller to handle marshalling and unmarshalling the data.

I chose to put all of these in my src/main/resources/applicationContext.xml.

Below is my entire applicationContext.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans   http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <import resource="classpath:META-INF/spring/batchContext.xml"/>

    <context:component-scan base-package="com.javaninja.batch"/>

    <bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>

    <!--This repository is only really intended for use in testing and rapid prototyping-->
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean"/>

    <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"
          p:jobRepository-ref="jobRepository"/>

    <bean id="jaxb2Marshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
        <property name="classesToBeBound">
            <list>
                <value>com.javaninja.batch.Car</value>
            </list>
        </property>
    </bean>

    <!-- ItemReader which reads data from XML file -->
    <bean id="xmlItemReader" class="org.springframework.batch.item.xml.StaxEventItemReader"
          p:resource="classpath:cars-input.xml"
          p:fragmentRootElementName="car"
          p:unmarshaller-ref="jaxb2Marshaller"/>

    <!-- XML ItemWriter which writes the data in XML format -->
    <bean id="xmlItemWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter"
        p:resource="file:xml/cars.xml"
        p:rootTagName="cars"
        p:marshaller-ref="jaxb2Marshaller"/>

</beans>

Spring Batch jobs are defined by fairly simple xml. You define a job, with one or more steps. In my case, I have one step. I defined it to use my reader and my writer and commit data in 1000 chunks. I placed this file in src/main/resources/META-INF/spring/batchContext.xml. Below is my entire batchContext.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:batch="http://www.springframework.org/schema/batch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd">

    <batch:job id="carJob">
        <batch:step id="step1">
            <batch:tasklet>
                <batch:chunk reader="xmlItemReader" writer="xmlItemWriter" commit-interval="1000"/>
            </batch:tasklet>
        </batch:step>
    </batch:job>

</beans>

XML Mapping

I used JAXB to map the XML objects. I chose a fairly simple example for demonstration purposes. I also chose to read and write the same XML, without modification.

The JAXB-annotated class is below.

package com.javaninja.batch;

import javax.xml.bind.annotation.XmlRootElement;

/**
 * @author norris.shelton
 */
@XmlRootElement
public class Car {
    private String make;
    private String model;
    private String color;
    private int doors;

    public String getMake() {
        return make;
    }

    public void setMake(String make) {
        this.make = make;
    }

    public String getModel() {
        return model;
    }

    public void setModel(String model) {
        this.model = model;
    }

    public String getColor() {
        return color;
    }

    public void setColor(String color) {
        this.color = color;
    }

    public int getDoors() {
        return doors;
    }

    public void setDoors(int doors) {
        this.doors = doors;
    }
}

Here is an example of the XML that I will be reading and writing.

<?xml version="1.0" encoding="UTF-8"?>
<cars>
    <car>
        <color>color1</color>
        <doors>1</doors>
        <make>make1</make>
        <model>model1</model>
    </car>
</cars>

Running a Job

Jobs are ran as a Java application. I defined my Main(). Inside, it instantiated an XML-context from the classpath. From there, I retrieved the JobLauncher and the Job that I defined in the batchContext.xml Then I ran the job.

The source is below.

package com.javaninja.batch;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobExecutionException;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

/**
 * @author norris.shelton
 */
public class Main {

    public static void main(String[] args) {
        ApplicationContext context = new ClassPathXmlApplicationContext("applicationContext.xml");

        JobLauncher jobLauncher = context.getBean("jobLauncher", JobLauncher.class);
        Job job = context.getBean("carJob", Job.class);

        try {
            JobExecution execution = jobLauncher.run(job, new JobParameters());
            System.out.println("Job Exit Status : " + execution.getStatus());

        } catch (JobExecutionException e) {
            System.out.println("Job ExamResult failed");
            e.printStackTrace();
        }
    }
}

When I ran the program, the output directory (xml) was written alongside the project directory, not inside.

Summary

In this blog, I showed you how to create a simple batch job that reads from an XML file and writes to an XML file. This was accomplished with very little business logic.

The entire project used to write this blog is located on GitHub sheltonn / spring-batch-xml

February 17th, 2016

Posted In: Java, java ninja, Javaninja, Spring Batch

2 Comments

LinkedIn Auto Publish Powered By : XYZScripts.com