Wednesday 27 May 2015

[Java / Guava] How to generate map while having either key or value ?

Map is a data structure which every software developer uses on a daily basis. Maps are generally created to provide a quick access to some object associated to a key. On the other hand we often create maps just have a more conveniant access to a collection of objects. Consider the following class as a main domain object:
public class Book {
 private final String title;
 private final String isbn;

 public Book(String title, String isbn) {
  this.title = title;
  this.isbn = isbn;
 }

 public String getTitle() {
  return title;
 }

 public String getIsbn() {
  return isbn;
 }
}
In case we want to retrieve a book of given isbn number from a list we would do something like this:
public class BookShelf {
    private static final List<Book> books = ImmutableList.of(
            new Book("Metro 2033", "54213452435"),
            new Book("The Witcher", "123123123")
    );

    public Book getBookByIsbn(String isbn) {
        for (Book book : books) {
            if (book.getIsbn().equals(isbn)) {
                return book;
            }
        }

        return null;
    }
}
Here's a simple unit test which checks whether the book has been found:
@Test
    public void shouldReturnMetro() throws Exception {
        // given
        String isbn = "54213452435";

        // when
        final Book book = new BookShelf().getBookByIsbn(isbn);

        // then
        assertThat(book.getTitle()).isEqualTo("Metro 2033");
    }
As you can see we have to iterate through the list and check whether book's isbn is equal to the one passed to the method. Such construction increases method complexity (if you use sonar or some similar tool you probably try to keep as lowest as it's possible). Other thing you have to take care of is a value returned in case the book of given isbn does't exist. I think I chose the worst possible solution which is returning null :) Other options may be:
  • -throwing exception
  • -null object pattern
Let's see how it would look if the books were stored in a map.
public class BookShelf {
    private static final Map<String, Book> books = ImmutableMap.of(
            "54213452435", new Book("Metro 2033", "54213452435"),
            "123123123", new Book("The Witcher", "123123123")
    );

    public Book getBookByIsbn(String isbn) {
        return books.get(isbn);
    }
}
I think it looks much better now. The unit test still passess and null value is returned by default when value for a given key doesn't exist. In real life such map won't exist Typically you fetch books from database or some restful service so it will be a list of books not the map. Let's see how to generate a map from the list in traditional imperative way:
public class BookShelf {
    private final Map<String, Book> books;

    public BookShelf(List<Book> books) {
        Map<String, Book> booksMap = Maps.newHashMap();
        for (Book book : books) {
            booksMap.put(book.getIsbn(), book);
        }

        this.books = booksMap;
    }

    public Book getBookByIsbn(String isbn) {
        return books.get(isbn);
    }
}
And the unit test:
@Test
    public void shouldReturnMetro() throws Exception {
        // given
        List<Book> books = ImmutableList.of(
                new Book("Metro 2033", "54213452435"),
                new Book("The Witcher", "123123123")
        );

        // when
        final Book book = new BookShelf(books).getBookByIsbn("54213452435");

        // then
        assertThat(book.getTitle()).isEqualTo("Metro 2033");
    }
Works fine... but I really don't like the way it's been done. Looks just ugly. Fortunately Guava can help here :) There's a uniqueIndex() method in Maps class which takes Iterable and Function as arguments.
public static <K, V> ImmutableMap<K, V> uniqueIndex(
      Iterable<V> values, Function<? super V, K> keyFunction) {
    return uniqueIndex(values.iterator(), keyFunction);
  }
The fcuntion will be applied to every book in Iterable and will generate a key so uniqueIndex() returns ImmutableMap in which value returned from function is a key and a book a value.
public class BookShelf {
    private final Map<String, Book> books;

    public BookShelf(List<Book> books) {
        this.books = Maps.uniqueIndex(books, new Function<Book, String>() {
            public String apply(Book book) {
                return book.getIsbn();
            }
        });
    }

    public Book getBookByIsbn(String isbn) {
        return books.get(isbn);
    }
}
Much better now. The Function may obviously be injected. In such case changing the function becomes extremely easy. Consider the following example in Spring:
@Component
public class BookShelf {
    private final Map<String, Book> books;

    @Autowired
    public BookShelf(ImmutableList<Book> books, Function<Book, String> bookIndexFunction) {
        this.books = Maps.uniqueIndex(books, bookIndexFunction);
    }

    public Book getBookByIsbn(String isbn) {
        return books.get(isbn);
    }
}
And the configuration:
@Configuration
@ComponentScan("gt.dev.sample")
public class BookshelfConfig {
    @Bean
    public Function<Book, String> bookIndexFunction() {
        return new Function<Book, String>() {
            public String apply(Book book) {
                return book.getIsbn();
            }
        };
    }

    @Bean
    public ImmutableList<Book> books() {
        return ImmutableList.of(
                new Book("Metro 2033", "54213452435"),
                new Book("The Witcher", "123123123")
        );
    }
}
It's just an example so I've injected the books as well. Let's run the application:
public class App {
    public static void main( String[] args ) {
        final BookShelf bookShelf = new AnnotationConfigApplicationContext(BookshelfConfig.class).getBean(BookShelf.class);
        final Book bookByIsbn = bookShelf.getBookByIsbn("54213452435");
        System.out.println(bookByIsbn);
    }
}
And the output is:
maj 26, 2015 1:42:46 PM org.springframework.context.annotation.AnnotationConfigApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@67eaf25d: startup date [Tue May 26 13:42:46 CEST 2015]; root of context hierarchy
Book{title=Metro 2033, isbn=54213452435}
Now let's say that I want to generate map while having only a list of keys. uniqueIndex() method cannot be used because it does the opposite so Guava Maps class contains following methods:
  • asMap()
  • toMap()
The only difference between those two methods is that the toMap() method returns an instance of ImmutableMap while asMap() returns a view of the original map. Now I've added the following bean into the configuration:
@Bean
    public ImmutableList<String> theWitcherIsbns() {
        return ImmutableList.of(
                "325252525", 
                "432653462", 
                "23463667", 
                "324632636");
    }
The list contains ISBN numbers of different editions of The Witcher book. I want to create a map in which the key is ISBN and the value an object of Book class. In traditional imperative way you would write another foreach loop which creates an object of Book class and puts it into the map which has to be created before the loop as well. Using Guava it can be written like that:
@Component
public class BooksGenerator {
    private final ImmutableMap<String, Book> theWitcherBooks;

    @Autowired
    public BooksGenerator(ImmutableList<String> theWitcherIsbns) {
        this.theWitcherBooks = Maps.toMap(theWitcherIsbns, new Function<String, Book>() {
            public Book apply(String isbn) {
                return new Book("The Witcher", isbn);
            }
        });
    }

    public Book getWitcherBookByIsbn(String isbn) {
        return theWitcherBooks.get(isbn);
    }
}
After starting the application:
public class App {
    public static void main( String[] args ) {
        final BooksGenerator booksGenerator = new AnnotationConfigApplicationContext(BookshelfConfig.class).getBean(BooksGenerator.class);
        final Book book = booksGenerator.getWitcherBookByIsbn("23463667");
        System.out.println(book);
    }
}
I get a proper book:
maj 26, 2015 4:53:27 PM org.springframework.context.annotation.AnnotationConfigApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@6f609af9: startup date [Tue May 26 16:53:27 CEST 2015]; root of context hierarchy
Book{title=The Witcher, isbn=23463667}
asMap() method works the same but as I've mentioned before it returns a view of original map. I strongly recommend using toMap() as immutable collections are more safe (threads) and it's harder to complicate the code using them.

Sunday 17 May 2015

[Bash / ssh] How to invoke command remotely without password / private key prompts ?

Sometimes you may want to invoke command on remote machine via ssh. You can obviously pass a command in double quotes:
ssh root@somehost.com "echo \$HOME"
This example prints to the console a value of env variable HOME (note that the dollar sign has to be escaped otherwise HOME variable will be resolved on your local machine). Let's say I want to fetch a value of some env variable in my bash script which will be started by Jenkins. There are actually two problems:
  • ssh will prompt for password,
  • if you haven't already accepted host's key there will be another prompt.
If you don't have proper entry in ~/.ssh/known_hosts you will see:
gt ~ ssh root@somehost.com "echo \$HOME"
                                                        
The authenticity of host 'somehost.com (10.92.30.38)' can't be established.
RSA key fingerprint is b0:c6:ad:6b:06:73:a3:de:31:8c:f8:4d:07:4e:2c:e6.
Are you sure you want to continue connecting (yes/no)? 

so you need to type "yes" in order to invoke the command.
In case you've already accepted the key you will see only:
gt ~ ssh root@somehost.com "echo \$HOME"                                                                                                           
root@somehost.com's password: 
Ssh doesn't have any flag for password (security) so you cannot do something like:
ssh root@somehost.com -p mySecretPassword
Solution for that is sshpass. I'm sure it's available in your linux distribution's repository. On Fedora install it using:
sudo yum install sshpass
So now you can pass the password easily:
gt ~ sshpass -p mySecretPassword ssh root@somehost.com "echo \$HOME"                                    
/root
In case you need to accept host's key you can use ssh -oStrictHostKeyChecking=no. Example:
gt ~ ssh root@somehost.com "echo \$HOME"                                                                          
The authenticity of host 'somehost.com (10.92.30.38)' can't be established.
RSA key fingerprint is b0:c6:ad:6b:06:73:a3:de:31:8c:f8:4d:07:4e:2c:e6.
Are you sure you want to continue connecting (yes/no)? ^C
zsh: interrupt  ssh root@somehost.com "echo \$HOME"
gt ~ sshpass -p mySecretPassword ssh -oStrictHostKeyChecking=no root@somehost.com "echo \$HOME"                             
Warning: Permanently added 'somehost.com,10.92.30.38' (RSA) to the list of known hosts.
/root
There is sctually another way of importing keys - ssh-keyscan command which output has to be appended to ~/.ssh/known_hosts file.
ssh-keyscan -H somehost.com >> ~/.ssh/known_hosts
Example:
gt ~ ssh root@somehost.com          
The authenticity of host 'somehost.com (10.92.30.39)' can't be established.
RSA key fingerprint is 3d:7d:a0:82:d7:3b:60:bc:58:ce:14:d2:bf:1e:d5:89.
Are you sure you want to continue connecting (yes/no)? ^C
zsh: interrupt  ssh root@somehost.com
gt ~ ssh-keyscan -H somehost.com >> ~/.ssh/known_hosts  
# somehost.com SSH-2.0-OpenSSH_5.3
# somehost.com SSH-2.0-OpenSSH_5.3
no hostkey alg
gt ~ ssh root@somehost.com 
Warning: Permanently added the RSA host key for IP address '10.92.30.39' to the list of known hosts.
root@somehost.com's password: 
Last login: Thu May 14 15:24:06 2015 from 10.154.8.71
[root@somehost ~]#
Value returned by the command invoked on remote host can obviously be assigned to some variable in bash script:
gt ~ cat script.sh
#!/bin/bash
REMOTE_HOME=$(sshpass -p arthur ssh -oStrictHostKeyChecking=no root@somehost.com "echo \$HOME")
echo "remote home = ${REMOTE_HOME}"
gt ~ ./script.sh
remote home = /root
As you can see both problems can be solved quite easily but you should realize that this kind of hacks (sshpass) shouldn't be used in production environment. Actually I use this kind of scripts which pass password in plain text only in test environments which aren't directly connected to the internet. Generally such machines are used only for snapshots' testing and don't store any crucial data. You should definitely read this part of sshpass man page:
SECURITY CONSIDERATIONS

First and foremost, users of sshpass should realize that ssh's insistance on only getting the password interactively is not without reason. 
It is close to impossible to securely store the password, and users of sshpass should consider whether ssh's public key authentication provides the same end-user experience, while involving less hassle and being more secure.

The -p option should be considered the least secure of all of sshpass's options. 
All system users can see the password in the command line with a simple "ps" command. Sshpass makes a minimal attempt to hide the password, but such attempts are doomed to create race conditions without actually solving the problem. 
Users of sshpass are encouraged to use one of the other password passing techniques, which are all more secure.

In particular, people writing programs that are meant to communicate the password programatically are encouraged 
to use an anonymous pipe and pass the pipe's reading end to sshpass using the -d option. 

Monday 27 April 2015

[Spring] How to create global exception handler ?

Sometimes you may want to handle exceptions globally. Let's say you have a service which exposes several rest endpoints which are conntected to the same database. Your data access objects may throw DatabaseConnectionException (or something like that) when db is not available. In such case the response should contain proper HTTP status code and error message. Sample rest endpoint:
@RestController
public class TemperatureEndpoint {
 @Inject
 private WeatherService temperatureService;

 @RequestMapping(
  value = "/temperature/{city}",
  method = RequestMethod.GET,
  produces = "application/json"
 )
 @ResponseBody
 public Temperature currentTemperature(@PathVariable("city") String city) {
  return temperatureService.getCurrentTemperatureIn(city);
 }
}
And another one:
@RestController
public class HumidityEndpoint {
 @Inject
 private HumidityService humidityService;

 @RequestMapping(
  value = "/humidity/{city}",
  method = RequestMethod.GET,
  produces = "application/json"
 )
 @ResponseBody
 public Humidity currentHumidity(@PathVariable("city") String city) {
  return humidityService.getCurrentHumidityIn(city);
 }
}
Both HumidityService and TemperatureService use the same database so you may use some unified runtime exception like DatabaseConnectionException. Handling the exception in both services isn't a good idea because the result will in most cases be the same. By the way each try-catch block makes you to write additional unit test so global handler seems to be a natural choice. Spring allows you to create a global exception handler (will be applied to all the rest controllers) using @ControllerAdvice annotation. Typically class annotated with ControllerAdvice contains methods which map exceptions to HTTP status codes and messages. In well-designed applications it can be the only place where exception handling occurs. Let's create some mappings. Each mapper should also generate proper json response so the client can handle it. Let's create a class annotated with @ControllerAdvice (note that the class has to be in scope of component scan).
@ControllerAdvice
public class WeatherServiceExceptionHandler {
        private static final Logger LOG = LoggerFactory.getLogger(WeatherServiceExceptionHandler.class);
}
I've added a logger because we may want to log something. Let's get back to DatabaseCommunicationException. In case the exception is thrown I want my service to log proper information and return exception's message in json so the client can handle it properly.
@ResponseStatus(value = HttpStatus.INTERNAL_SERVER_ERROR)
@ExceptionHandler(DatabaseCommunicationException.class)
public void handleDatabaseConnectionException(DatabaseCommunicationException e) { 
 LOG.error("DatabaseCommunicationException occurred", e);
}
As you can see I've created a method which takes as a parameter DatabaseCommunicationException so I can log its content. @ExceptionHandler annotation tells Spring that the method handles DatabaseCommunicationException exception. @ResponseStatus allows you to specify which HTTP status will be assigned to the response when this particular handler is being fired. In this case WeatherService cannot work without database connection so I chose 500 which is internal server error. You can obviously use whichever code you want to. For now the handler only logs a message and returns http code but it should also return a message to the client. In all RestControllers I use jackson to map POJO cleassess to json. It can be used in ControllerAdvice as well. Jackson dependency:
<dependency>
 <groupId>com.fasterxml.jackson.core</groupId>
 <artifactId>jackson-databind</artifactId>
</dependency>
In this example json response will only contain a message but you can add here anything else. My response is just a sample immutable POJO. It looks as follows:
public class ErrorMessage {
 private final String message;

 public ErrorMessage(String message) {
  this.message = message;
 }

 public String getMessage() {
  return message;
 }
}
Now the handler should return ErrorMessage and indicate that response should be converted into json format. Complete method looks like this:
@ResponseStatus(value = HttpStatus.INTERNAL_SERVER_ERROR)
@ExceptionHandler(DatabaseCommunicationException.class)
@ResponseBody
public ErrorMessage handleDatabaseConnectionException(DatabaseCommunicationException e) {
 LOG.error("DatabaseCommunicationException occurred", e);
 return new ErrorMessage(e.getMessage());
}
Let's add another one. This time I want to tell the user that the service doesn't support requested city. Fo instance the user wants to get current temperature in Goszowice (which is very small village in southern Poland) but the database simply doesn't contain any data about this place. In such case TemperatureService throws UnsupportedCityException. I guess HttpStatus.NOT_FOUND (404) fits best.
@ResponseStatus(value = HttpStatus.NOT_FOUND)
@ExceptionHandler(UnsupportedCityException.class)
@ResponseBody
public ErrorMessage handleUnsupportedCityException(UnsupportedCityException e) {
 LOG.error("UnsupportedCityException occurred", e);
 return new ErrorMessage(e.getMessage());
}
As you can see bodies of both handlers are almost the same so it can be extracted to new method. It's just an example so let it be as it is. Complete code:
@ControllerAdvice
public class WeatherServiceExceptionHandler {
 private static final Logger LOG = LoggerFactory.getLogger(WeatherServiceExceptionHandler.class);

 @ResponseStatus(value = HttpStatus.INTERNAL_SERVER_ERROR)
 @ExceptionHandler(DatabaseCommunicationException.class)
 @ResponseBody
 public ErrorMessage handleDatabaseConnectionException(DatabaseCommunicationException e) {
  LOG.error("DatabaseCommunicationException occurred", e);
  return new ErrorMessage(e.getMessage());
 }

 @ResponseStatus(value = HttpStatus.NOT_FOUND)
 @ExceptionHandler(UnsupportedCityException.class)
 @ResponseBody
 public ErrorMessage handleUnsupportedCityException(UnsupportedCityException e) {
  LOG.error("UnsupportedCityException occurred", e);
  return new ErrorMessage(e.getMessage());
 }
}
You can obviously provide a better way of constructing error message because you may not want to expose your internal messages. Something like this looks a bit better:
private static final ImmutableMap<Class, String> EXCEPTION_TO_MESSAGE_MAP = ImmutableMap.<Class, String>of(
   DatabaseCommunicationException.class, "Service is not available. Please try again later.",
   UnsupportedCityException.class, "Service does not track temperature in requested city."
);
That would be all :) Cheers

Friday 10 April 2015

[Spring] How to test RestController using MockMvc ?

If you work in a software company you probably write integration tests which deploy your product into application server and then test functionalities exposed by web services. Integration tests ran by continuous integration server are very useful. Especially when the whole team develops some kind of functionality. After each commit you can easily check whether everything works fine and continue the development. Unfortunately it takes time... Typical workflow may look like this one: 1. Developer commits the code. 2. The code is being built on CI server. 3. Test environment has to be upgraded (war deployment). 4. The tests. It strongly depends on how big your project is but I'm sure you will have to wait for a while. Perfect tests should work as fast as unit tests. You should be able to run whole test suite and see results after seconds. And here comes Spring. Each rest controller can be conveniently tested using MockMvc. In order to do that you need to add spring-test into your dependencies' section:
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-test</artifactId>
  <scope>test</scope>
</dependency>
Let's say I want to test following endpoint:
@RestController
public class WeatherEndpoint {
  @Inject
  private WeatherService weatherService;
  
  @RequestMapping(value = "/weather/{city}",
                    method = RequestMethod.GET,
                    produces = "application/json"
  @ResponseBody
  public Weather currentWeather(@PathVariable("city") String city) {
    return weatherService.getCurrentWeatherIn(city);                                    }
}
It's just an example so my WeatherService returns fixed value:
private class WeatherService {
  public Weather getCurrentWeatherIn(String city) {
    return new Weather("20.3");
  }
}

@Bean
public WeatherService weatherService() {
  return new WeatherService();
}
We need to prepare test skeleton which will be able to: 1. Run the test using spring runner. 2. Run application context. 3. Expose instance of MockMvc class.
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = {AppConfig.class})
@WebAppConfiguration
public class WeatherEndpointTest {
    private MockMvc mockMvc;

    @Inject
    private WebApplicationContext context;

    @Before
    public void contextSetup() {
        this.mockMvc = mockMvc();
    }

    private MockMvc mockMvc() {
        return MockMvcBuilders.<StandaloneMockMvcBuilder>webAppContextSetup(context).build();
    }
}
Now I'm going to explain each fragment of this template. @RunWith(SpringJUnit4ClassRunner.class) - indicates that Spring runner will be used. Unfortunately JUnit test can use only one runner so forget about Mockito runner. @ContextConfiguration(classes = {AppConfig.class}) - the runner will setup spring context using configuration classes listed in this annotation. In this example it will use only AppConfig. private MockMvc mockMvc - instance of MockMvc. This object allows to perform operations on rest endpoints. @Inject private WebApplicationContext context - context has to be injected in order to build mockMvc contextSetup() and mockMvc() - before each test mockMvc will be rebuilt Basically that's all. You can now start writing tests. Note that you can inject beans which have been added to spring container. In next episode I will explain how to do that and how to mock some of them in order to make tests isolated. currentWeather() returns Weather object which is a POJO class that contains only one field - (String) temperature. I've added jackson as dependency so the instance of Weather is being mapped to json. Here's our test:
    @Test
    public void shouldReturnInfoAboutWeather() throws Exception {
        // given
        String city = "Wroclaw";

        // when // then
        mockMvc.perform(get("/weather/" + city))
                .andExpect(status().isOk())
                .andExpect(content().contentType(MediaType.APPLICATION_JSON))
                .andExpect(jsonPath("temperature").value("20.3"));
    }
If you run the test you will see something like: INFO: Mapped "{[/weather/{city}],methods=[GET],params=[],headers=[],consumes=[],produces=[application/json],custom=[]}" onto public gt.dev.mockmvc.WeatherEndpoint.currentWeather(java.lang.String) which means that the context has been run and /weather endpoint exposed. In this particular example MockMvc performs GET on the endpoint and checks the result. In this case I expect http status to be OK (200) and media type json. Additionally I've added:
  <dependency>
   <groupId>com.jayway.jsonpath</groupId>
   <artifactId>json-path</artifactId>
   <scope>test</scope>
  </dependency>
so I can test the result mapped to json. Here I check whether temperature is equal to 20.3 (fixed value). Actually json-path is a really cool tool. You can assert that your json response is valid using very readable methods. The whole test below:
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = {AppConfig.class})
@WebAppConfiguration
public class WeatherEndpointTest {
    private MockMvc mockMvc;

    @Inject
    private WebApplicationContext context;

    @Before
    public void contextSetup() {
        this.mockMvc = mockMvc();
    }

    @Test
    public void shouldReturnInfoAboutWeather() throws Exception {
        // given
        String city = "Wroclaw";

        // when // then
        mockMvc.perform(get("/weather/" + city))
                .andExpect(status().isOk())
                .andExpect(content().contentType(MediaType.APPLICATION_JSON))
                .andExpect(jsonPath("temperature").value("20.3"));
    }

    private MockMvc mockMvc() {
        return MockMvcBuilders.<StandaloneMockMvcBuilder>webAppContextSetup(context).build();
    }
}
I strongly recommend to experiment with mockMvc. You should definitely check what can be passed to andExpect() method. That's all folks. Next time I'm going to show how to mock services in sping container in order to make integration tests isolated.