1/24/12

Advanced integration testing with Pax Exam Karaf

Prologue
From my first steps in programming, I realized the need of integration testing.

I remember setting up a macro recorder in order to automate the tests of my first swing applications. The result was a disaster, but I think the idea was in the right direction even though the mean was wrong.

Since then I experimented with a lot of tools for unit testing, which I think that are awesome, but I always felt that they were not enough for testing my "actual product".

The last couple of years I have been working with integration projects and also with OSGi and there the challenge of testing the "actual product" is bigger. The scenarios that need to be tested sometimes involve more than one VMs and of course testing OSGi is always a challenge.

In this post I am going to blog about Pax Exam Karaf, which is an integration testing framework for Apache Karaf and also the answer to all my prayers. Kudos to Andreas Pieber for creating it.

Tools for integration tests in OSGi
I have been using Pax Exam for a while now for integration tests in a lot of project I have been working on (e.g. Apache Camel). Even though its a great tool it does not provide the "feel" that the tests run inside Apache Karaf but rather inside the underlying OSGi container.

A tool which is not a testing framework itself, but is frequently used for OSGi testing is PojoSR. It actually allows you to use the OSGi service layer without using the module layer (if you prefer its an OSGi service registry that run without an OSGi container). So its sometimes used for testing OSGi services etc. A great example is the work of Guillaume Nodet (colleague of mine at  FuseSource and yes I know that he doesn't need introduction) for testing Apache Camel routes that use the OSGi blueprint based on PojoSR. You can read more about it in Guillaume's post. Very powerful but yet it is not inside Karaf (it tests the routes in blueprint, but it doesn't test OSGi stuff).

Arqullian is an other effort that among others allow you to run OSGi integration tests but I haven't used that myself.


Pax Exam Karaf
Each of the tools mentioned in the previous section are great and they all have their uses. However, none of the above gives me the "feel" that the tests do run inside Apache Karaf.

Pax Exam Karaf is a Pax Exam based framework which allows you to run your tests inside Apache Karaf. To be more accurate it allows you to run your tests inside any karaf based container. So it can also be Apache ServiceMix, FuseESB or even a custom runtime that you have created yourself on top of Apache Karaf.

As of Karaf 3.0.0 (it's not yet released) it will be shipped as part of Karaf tooling. Till then you can enhoy it at the paxexam-karaf github repository.


The benefits it offers over traditional approaches is that it allows you to use all Karaf goodness inside your tests:
  • Features Concept
  • Deployers
  • Admin Service
  • Dynamic configuration
  • more
Setting it up
Detailed installation instructions can be found at the paxexam-karaf wiki.

The basic idea is that your unit test is added inside a bundle called probe and this bundle is deployed inside the container. The container can be customized, with custom configuration, custom features installation etc so that it fits the "actual" environment.

Starting a custom container
To setup the container of your choiece you can just modify the configuration method. Here is an example that uses Apache ServiceMix as a target container:


Using the Karaf shell
Since one of Karaf's awsome features is its shell, being able to use shell commands inside an integration test is really important.


The first thing that needs to be addressed in order to use the shell inside the probe, is that the probe will import all the required packages. The probe by default uses a dynamic import on all packages, however this will exclude all packages exported with provisional status. To understand what this means you can read more about the provisional osgi api policy. In our case we need to customize our probe and allow it to import such packages this can be done by adding the following method to our test:


Now in order to execute commands you need to get the Command processor from the OSGi service registry and use it to create a session. Below is a method that allows to do in order to execute multiple commands under the same session (this is useful when using stateful commands like config). Moreover this method allows you to set a timeout on the command execution.



Distributed integration tests in Karaf
Working full time on open source integration, my needs often spawn across the boundaries of single JVM. A simplistic example is having one machine sending http requests to an other, having multiple machines exchanging messages through a message broker, or even having gird. The question is how do you test these cases automatically.

Karaf provides the admin service, which allows you to create and start instances of Karaf that run inside a separate jvm. Using it from inside the integration test you are able to start multiple instances of Karaf and deploy to each instances the features/bundles that are required for your testing scenario.

Test Scenario:
Let's assume that you want test a scenario where you have on jvm that acts as a message broker, one jvm that acts as a camel message producer and one jvm that acts as a camel message consumer. Also let's assume that you don't run vanilla karaf, but FuseESB (which is an Enterprise version of ServiceMix, which is powered by Karaf). You can use the admin service from inside your integration test just like this.


You might wonder now how are you supposed to assert that the consumer actually consumed the messages that it was supposed to. There are a lot of things that could be done here:

  • Connect to consumer node and use camel:route-info
  • Connect to the consumer and check the logs (if its supposed to log something)
  • Get the jmx stats of the broker

You can literally do whatever you like.

So here's how it would like if we go with using the camel commands:


Pointers to working examples
I usually provide sources for the things I blog. In this case I'll prefer to point you to existing OSS projects that I have been working on and are using paxeaxam-karaf for integration testing.

Fuse Fabric integration tests: Contains test that check the installation of features, availability of services, distributed OSGi services, even the creation and use of agents in the cloud.

Jclouds-Karaf integration tests: Contains simple tests that verifies that features are installed correctly.

Apache Karaf Cellar integration tests: Contains distributed tests, with quite a few Hazelcast examples.

24 comments:

  1. Excellent article. Exactly what I was looking for!

    ReplyDelete
  2. Thanks a lot! I am really glad that you found it useful!

    ReplyDelete
  3. Thanks for this useful article. Is it also possible to create distributed integration test on remote hosts? (So it is possible to use specific hardware between different hosts.)

    ReplyDelete
  4. Not directly. By this I mean that Karaf itself doesn't provide direct means of creating remote instances.

    At FuseSource I am also working on an project that can help you solve your problem: http://fabric.fusesource.org/. Fabric among others will allow you to install Karaf containers on any host you have ssh access. That can be really helpful in your tests. Here is an example: https://github.com/fusesource/fuse/blob/master/fabric/fabric-itests/fabric-pax-exam/src/test/java/org/fusesource/fabric/itests/paxexam/CreateSshAgentTest.java

    ReplyDelete
    Replies
    1. Thanks! Great to see the possibilities with Fuse Fabric. I investigate what the best solution is for us.

      Delete
  5. Thanks for this useful article. Is it possible to execute these commands on remote hosts "ssh port of karaf"? or we have to execute it directly in the same JVM as karaf instance ?

    ReplyDelete
  6. Thanks for your words!

    First allow me to clarify, that instances created by the admin service DO NOT run in the same JVM, but they do run on the same host.

    If you have an existing Karaf/ServiceMix/FuseESB or what not running on a remote host an you want to execute commands on the remote host, you can use Karaf's ssh command:

    karaf@root> ssh -p remoteHostPort remoteHostAddress command

    Now, if you want to install Karaf on a remote host as part of your test, that's not directly supported by Karaf. That kind of functionality is provided by Fuse Fabric: http://fabric.fusesource.org. Fabric will allow you to spin up container to remote hosts via ssh or jclouds if you are interesting in adding cloud into the equation.

    I hope this helps.

    ReplyDelete
  7. Hi Ioannis thanks for this article, very useful indeed. One question - in the executeCommands method above, where are you getting your executor from?

    Thanks again, keep up the good work!

    ReplyDelete
  8. I just create an executor and issue it. I don't think that for testing we need something more sophisticated than this one!
    Thanks for your kind words!

    ReplyDelete
  9. This post is probably where I got the most useful information for my research. Thanks for posting, maybe we can see more on this.
    Are you aware of any other websites on this
    testing-tools

    .

    ReplyDelete
    Replies
    1. @Robert, thanks! I intend to get back to the subject in the near future.
      Nothing comes to mind right now.

      Delete
  10. Have you tried any tool that reports code coverage for the integration tests with Pax Exam Karaf?

    ReplyDelete
  11. Hey Friend,

    Greetings to all of you. I am happy after reading your post. I noticed some useful tips from this post. Thanks for sharing this.


    Best Regards,

    Test Probes

    ReplyDelete
  12. Hello,

    First off thank you for your post, it has been very useful.
    Secondly I have a question.

    When I try to add the CommandProcessor and I use probe.setHeader(Constants.DYNAMICIMPORT_PACKAGE, "*;status=provisional"); I get the following stack trace:

    java.lang.ClassNotFoundException: org.ops4j.pax.exam.options.MavenUrlReference not found by PAXEXAM-PROBE-8d3137f6-cc17-4f50-96cb-71e083fa3a71 [75]
    at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
    at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
    at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2442)
    at java.lang.Class.privateGetPublicMethods(Class.java:2562)
    at java.lang.Class.getMethods(Class.java:1427)
    at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.findAndInvoke(JUnitProbeInvoker.java:89)
    at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.call(JUnitProbeInvoker.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.ops4j.pax.exam.rbc.internal.RemoteBundleContextImpl.remoteCall(RemoteBundleContextImpl.java:86)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$1.run(Transport.java:177)
    at sun.rmi.transport.Transport$1.run(Transport.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

    Also I try to load the org.apache.felix.gogo.runtime artifact which has the package and code for the CommandProcessor. If not for it I get ClassNotFoundException when Karaf tries to inject the service.

    Do you have any suggestions what I should do?

    Thank you very much

    ReplyDelete
  13. Link not found: https://github.com/openengsb/labs-paxexam-karaf

    ReplyDelete
  14. @Bill: The project has been merged with Karaf 2.3.x itself. Also a part of it is moving to pax-exam.

    ReplyDelete
  15. @Pavel: Could you please provide some more info about the versions you are using (Karaf, Pax-Exam etc)?

    ReplyDelete
  16. Please get the history right; Pax Exam was "created" primarily by Toni Menzel, in collaboration with Alin Dreghiciu, in 2008. Toni was the shepherd for a very long time, but over the last couple of years many other people have stepped up and taken the project further. However, the history wasn't preserved appropriately when moving OPS4J from its own Subversion repository to GitHub...

    Thanks - Niclas

    ReplyDelete
  17. Hi,
    thank you for your post!
    i have the same error as Paval:
    java.lang.ClassNotFoundException: org.ops4j.pax.exam.options.MavenUrlReference not found by PAXEXAM-PROBE-86b7fa97-9d8d-491d-a931-029fc44fa36f [74]
    at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:787)
    at org.apache.felix.framework.ModuleImpl.access$400(ModuleImpl.java:71)
    at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1768)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
    at java.lang.Class.privateGetPublicMethods(Class.java:2547)
    at java.lang.Class.getMethods(Class.java:1410)
    at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.findAndInvoke(JUnitProbeInvoker.java:81)
    at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.call(JUnitProbeInvoker.java:72)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.ops4j.pax.exam.rbc.internal.RemoteBundleContextImpl.remoteCall(RemoteBundleContextImpl.java:80)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
    at sun.rmi.transport.Transport$1.run(Transport.java:159)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

    versions of pax exam and karaf:

    org.apache.karaf
    apache-karaf
    2.3.2
    zip
    test


    org.ops4j.pax.exam
    pax-exam-junit4
    3.1.0
    test



    I use this pax version because i need this useful feature install:
    org.ops4j.pax.exam.karaf.options.KarafDistributionOption.features(String repositoryUrl, String... features)
    and it is only available at 3.1.0

    Thank you

    ReplyDelete
  18. the problem has been resolved by adding proper dynamic import header.

    one more question, is there any means to run test after some timeout?
    thank you

    ReplyDelete
  19. Dennis, how did you do that exactly? I added

    @ProbeBuilder
    public TestProbeBuilder probeConfiguration(TestProbeBuilder probe)
    {
    probe.setHeader(Constants.DYNAMICIMPORT_PACKAGE, "*");
    return probe;
    }

    but without any success...

    ReplyDelete
  20. Benjamin,
    this works for me
    @ProbeBuilder
    public TestProbeBuilder probeConfiguration(TestProbeBuilder probe) {
    probe.setHeader(Constants.DYNAMICIMPORT_PACKAGE, "*,org.apache.felix.service.*;status=provisional");
    return probe;
    }

    ReplyDelete
  21. Hi, Canellos:

    A great description !

    Recently I met a trouble in karaf testing and the Pax Exam complains it can not find my custom service in another bundle, and I tried copy that service and paste it into my testing bundle and the problem still exist. Do you mind give me some hints ? It cost me 2 days in this test because the web searching can really give out few result about this trouble.


    And I posted it on stackoverflow, afaik nobody replay. That's sad because I really need someone help me in my urgent project, and I'll be really thankful about it, thank you at first!

    And ,the post in stackoverflow is here: http://stackoverflow.com/questions/20301589/pax-exam-karaf-container-inject-custom-service-encounters-classnotfoundexcepti

    ReplyDelete
  22. May you provide more information about the executor? thank you.

    ReplyDelete