May 09 2013

Test data – part 1

Posted by: MarcusPhilip @ 12:00

When you run an integration or system test, i.e. a test that spans one or more logical or physical boundaries in the system, you normally need some test data, as most non­trivial operations depends on some persistent state in the system. Even if the test tries to follow the advice of favoring to verify behavior over state, you may still need specific input to even achieve a certain behavior. For example, if you want to test an order flow for a specific type of product, you must know how to add a product of that type to the basket, e.g. knowing a product name.

But, and here is the problem, if you don’t have strict control of that data it may change over time, so suddenly your test will fail.

When unit testing, you’ll want to use mocks or fakes for dependencies (and have well factored code that lets you easily do that), but here I’m talking about tests where you specifically want to use the real dependency.

Basically, there are only two robust ways to manage test data:

  1. Each tests creates the data it needs.
  2. Create a managed set of data that covers all of your test needs.

You can also use a combination of the two.

For the first strategy, either you have an idempotent approach so that you just ensure a certain state, or, you create and delete the data for each run. In some cases you can use transactions to be able to safely parallelize your tests and not modify persistent state. Just open one at the start of the test and then abort it instead of committing at the end. Obviously you cannot test functionality that depends on transactions this way.

The second strategy is a lot easier if you already have a clear separation between reference data, application data and transactional data.

By reference data I mean data that change with very low frequency and that often is of limited size and has a list or key/value structure. Examples could be a list of supported languages or zip code to address lookup. This should be fairly easy to keep in one authoritative, version controlled location, either in bulk or as deltas.

The term application data is not as established as reference data. It is data that affects the behavior of the application. It is not modified by normal end user actions, but is continuously modified by developers or administrators. Examples could be articles in a CMS or sellable products in an eCommerce website. This data is crucial for tests. It’s typically the data that tests use as input or for assertions.

The challenge here is to keep the production data and the test data set in synch. Ideally there should be a process that makes it impossible (or at least hard) to update the former without updating the second. However, there are often many complicating factors: the data can be in another system owned by another team and without a good test double, the data can be large, or it can have complex relationships or dependencies that sometimes very few fully grasp. Often it is managed by non­technical people so their tool set, knowledge and skills are different.

Unit or component tests can often overcome these challenges by using a strategy to mock systems or create arbitrary test data and verify behavior and not exact state, but acceptance tests cannot do that. We sometimes need to verify that a specific product can be ordered, not a fictional one created by the test.

Finally, transactional data is data continuously created by the application. It is typical large, fast growing and of medium complexity. Example could be orders, article comments and logs.

One challenge here is how to handle old, ‘obsolete’ data. You may have data stored that is impossible to generate in the current application because the business rules (and the corresponding implementation) have changed. For the test data it means you cannot use the application to create the test data if that was you strategy. Obviously, this can make the application code more complicated, and for the test code, hopefully you have it organized so it’s easy to correlate the acceptance test to the changed business rule and easy to change them accordingly. The tests may get more complicated because there can now e.g. be different behavior for customers with an ‘old’ contract. This may be hard for new developers in the team that only know of the current behavior of the app. You may even have seemingly contradicting assertions.

Another problem can be the sheer size. This can be remediated by having a strategy for aggregating, compacting and/or extracting data. This is normally easy if you plan for it up front, but can be hard when your database is 100 TB. I know that hardware is cheap, but having a 100 TB DB is inconvenient.

The line between application data and transactional data is not always clear cut. For example when an end user performs an action, such as a purchase, he may become eligible for certain functionality or products, thus having altered the behavior of the application. It’s still a good approach though to keep the order rows and the customer status separated.

I hope to soon write more on the tougher problems in automated testing and of managing test data specifically.

Marcus Philip
@marcus_phi

Tags: , ,

Dec 21 2012

Writing integration tests with an in-memory Mongo DB

Posted by: TommyTynjä @ 16:42

As I mentioned in my previous post, I’ve been working closely to the document oritented NoSQL database Mongo DB lately. As an advocate of sustainable systems development (test driven development that is), I took the lead in our team for designing tests for our business logic towards Mongo. For relational databases there are a lot of options, a common solution for testing against relational databases is to use an in-memory database such as H2. For NoSQL databases the options are not always as generous from an automated test perspective. Luckily, we found an in-memory version for Mongo DB from Flapdoodle which is easy to use and fits our use case perfectly. If your (Java) code base relies on Maven, just add the following dependency:

<dependency>
    <groupId>de.flapdoodle.embed</groupId>
    <artifactId>de.flapdoodle.embed.mongo</artifactId>
    <version>1.27</version>
    <scope>test</scope>
</dependency>

Then, in your test class (here based on JUnit), use the provided classes to start an in-memory Mongo DB, preferrably in a @Before method, and tear down the instance in a @After method, as in the example below:

public class TestInMemoryMongo {

    private static final String MONGO_HOST = "localhost";
    private static final int MONGO_PORT = 27777;
    private static final String IN_MEM_CONNECTION_URL = MONGO_HOST + ":" + MONGO_PORT;

    private MongodExecutable mongodExe;
    private MongodProcess mongod;
    private Mongo mongo;

    /**
     * Start in-memory Mongo DB process
     */
    @Before
    public void setup() throws Exception {
        MongodStarter runtime = MongodStarter.getDefaultInstance();
        mongodExe = runtime.prepare(new MongodConfig(Version.V2_0_5, MONGO_PORT, Network.localhostIsIPv6()));
        mongod = mongodExe.start();
        mongo = new Mongo(MONGO_HOST, MONGO_PORT);
    }

    /**
     * Shutdown in-memory Mongo DB process
     */
    @After
    public void teardown() throws Exception {
        if (mongod != null) {
            mongod.stop();
            mongodExe.stop();
        }
    }

    @Test
    public void shouldAssertSomeInteractionWithMongo() {
        // Create a connection to Mongo using the IN_MEM_CONNECTION_URL property
        // Execute some business logic towards Mongo
        // Assert the expected the behaviour
    }

    protected MongoPersistenceContext getMongoPersistenceContext() {
        // Returns an instance of the class containing business logic towards Mongo
        return new MongoPersistenceContext();
    }
}

… then you have an in-memory Mongo DB available for your test cases. The private member named “mongo” in the example above is of type com.mongodb.Mongo, which you can use for interaction with the Mongo database. As you notice, all you need is basically JUnit och Mongo, nothing more. But life is not always as easy as the simplest examples. In our case, we leverage from EJBs and other components which relies on running inside an container. As I’m contributor to the JBoss Arquillian project, a Java framework for integration tests, I was obviously curious about trying the approach of making the in-memory available when executing tests inside the container. If you use Arquillian already like we do, the transition is smooth. Just make sure to put the in-memory Mongo and Java driver on your classpath. The following Arquillian test extends the JUnit test class above, but with a bean injection for the business class under test:

@RunWith(Arquillian.class)
public class TestInMemoryMongoWithArquillian extends TestInMemoryMongo {

    @Deployment
    public static Archive getDeployment() {
        return ShrinkWrap.create(WebArchive.class, "mongo.war")
                .addClass(MongoPersistenceContext.class)
                .addAsManifestResource(EmptyAsset.INSTANCE, "beans.xml")
                .addAsWebInfResource(new StringAsset("<web-app></web-app>"), "web.xml")
                .addAsLibraries(DependencyResolvers.use(MavenDependencyResolver.class)
                        .artifact("de.flapdoodle.embed:de.flapdoodle.embed.mongo:jar:1.27")
                        .artifact("org.mongodb:mongo-java-driver:jar:2.9.1")
                        .resolveAs(JavaArchive.class));
    }

    @Inject MongoPersistenceContext mpc;

    @Override
    protected MongoPersistenceContext getMongoPersistenceContext() {
        return mpc;
    }
}

I’ve uploaded a full example project, based on Java 7, Maven, Mongo DB, Arquillian and Apache TomEE (embedded) on GitHub to demonstrate how to use an in-memory Mongo DB in a unit test as well as with Arquillian inside TomEE. This should serve as a good base when starting to write automated tests for your business logic towards Mongo.


Tommy Tynjä
@tommysdk

Tags: , , , , , ,

Dec 19 2012

Applying indexes in Mongo DB

Posted by: TommyTynjä @ 11:14

For the past year I’ve been consulting at a client where we’ve been using the document oriented NoSQL database Mongo DB in production (currently v2.0.5). Primarily we store PDF documents together with some arbitrary metadata, but in some use cases we also store a lot of completely dynamic documents, where there might be no similar columns shared between documents in the same collection.

For one of our collections, which holds PDF documents in a GridFS structure (~ 1 TB of data/node), we sometimes need to query for documents based on a couple of certain keys. If these keys are not indexed, queries can take a very long time to execute. How indexes are handled are very well explained in the Mongo documentation. Per default, GridFS provides us indexes for the _id, filename + uploadDate fields. To view indexes on the current collection, execute the following command from the Mongo shell:

db.myCollection.getIndexes();

Ideally, you want your indexes to reside completely within RAM. The following command returns the size of the current index:

db.myCollection.totalIndexSize();

To apply a new index for keyX and keyY, execute the following command:

db.myCollection.ensureIndex({"keyX":1, "keyY":1});

Applying the index took roughly about a minute per node in our environment. The index should be displayed when executing db.myCollection.getIndexes();

{
        "v" : 1,
        "key" : {
                "keyX" : 1,
                "keyY" : 1
        },
        "ns" : "myDatabase.myCollection",
        "name" : "keyX_1_keyY_1"
}

After applying the index, assure that the total size of the index is still managable (less than avaiable memory). Now, executing a query based on the indexed fields should yield its result much faster:

db.myCollection.find({"keyX":9281,"keyY":3270});
Tommy Tynjä
@tommysdk

Tags: ,

Dec 06 2012

Use of generics in EJB 3.1

Posted by: TommyTynjä @ 22:46

With EJB 3.1 it is possible to make use of generics in your bean implementations. There are some aspects that needs to be considered though to make sure you use them as intended. It requires basic understanding of how Java generics work.

Imagine the following scenario. You might want to have a business interface such as:

@Local(Async.class)
public interface Async<TYPE> {
    public void method(TYPE t);
}

… and an implementation: 

@Stateless
public class AsyncBean implements Async<String> {
    @Asynchronous
    public void method(final String s) {}
}

You would then like to dependency inject your EJB into another bean such as:

@EJB Async<String> asyncBean;

A problem here is that due to type erasure, you cannot be certain that the injected EJB is of the expected type which will produce a ClassCastException in runtime. Another problem is that the method annotated with @Asynchronous in the bean implementation will not be invoked asynchronously as might be expected. This is due to the method not actually being part of the business interface. The method in the business interface is in fact public void method(final Object s) due to type erasure, which in turn calls the typed method. Therefore the call to the typed method won’t be intercepted and made asynchronous. If the type definition in the AsyncBean class is removed (infers Object type), the call will be asynchronous. One solution to ensure type safety would be to treat the Async interface as a plain interface without the @Local annotation, mark the AsyncBean with @LocalBean (to make sure the bean method is part of the exposed business methods for that bean), and inject an AsyncBean where used. Another solution would be to insert a local interface between the plain interface and the bean implementation:

@Local(AsyncLocal.class)
public interface AsyncLocal extends Async<String> {
    @Override
    public void method(final String s);
}

… and then make the AsyncBean implement the AsyncLocal interface instead, and use the AsyncLocal interface at the injection points.

This behaviour might not be all that obvious, even though it makes perfect sense due to how generics are treated in Java. It is also unfortunate that the EJB specification does not clarify the expected behaviour for this use case. The following can be read in the specification, which feels a bit vague:
“3.4.3 Session Bean’s Business Interface
The session bean’s interface is an ordinary Java interface. It contains the business methods of the session bean.”

I’ve created a minimal sample project where the behaviours can be tried out by just running a simple test case written with Arquillian. The test case will be run inside an embedded TomEE container. Thanks to the blazing speed of TomEE, the test runs just as fast as a unit test. The code can be found here, and contains a failing test representing the scenario described above. Apply the proposed soultion to make the test pass.

As demonstrated, there are some caveats which needs to be considered when using generics together with EJBs. Hopefully this article can shed some light on what is happening under the hood.

Tommy Tynjä
@tommysdk

Tags: , ,

Sep 21 2012

The importance of integration testing

Posted by: TommyTynjä @ 07:36

I’ve been studying integration testing quite close for the past year, especially since being a contributor to the JBoss Arquillian and ShrinkWrap projects, which target automated enterprise integration testing. This is a part which is almost non-existent in the enterprise today. Sure, if you have > 95% code coverage by unit tests you can be pretty certain that the business logic behaves like it is supposed to. But you are still testing your code in the wrong context. You still need to test your code in your target environment, e.g. how does the code actually behave when run in your application server, with a specific Java version, etc?

Unit tests are great and should be the cornerstone for each application that is developed today. But unit tests alone are not enough. You also need to test your code in the proper context, in the context where it is going to live in for a far longer time than it took to implement and release the first version. You need to test your code in the same environment that is going to be used in production. Why? Imagine a Formula 1 team creating a new aerodynamic part. If their simulations return promising results, they evaluate the part in the wind tunnel. But if the wind tunnel tests turn out good, they don’t put the part onto their race car five minutes before the start of the next race. It sounds like a silly thing to do, but that is exactly what is going on in system development today! The Formula 1 teams test new parts on the test track for days, in the right context – with wind, humidity, heat, stress, endurance, etc. Every parameter playing its part for the new detail that is being tested. Then they carefully analyze the results, before they decide whether this new component fits into the production environment (which is on the car during a Grand Prix weekend).

It’s when we connect pieces together, cogwheels to moving parts, when it gets interesting. How are the pieces working together? Do the cogwheels fit together at all? Is there friction which causes overheating? Can the pieces survive high torque? This is something that is extremely important also in software development, but is often neglected. I’ve been in projects where large teams works in two, three or four week sprints, but at the end of the sprint when all new code is integrated, the artifact doesn’t even deploy! It then usually takes a couple of developers two days just to integrate the code to make it deployable. And when it finally deploys, it doesn’t say anything about if the code works or not. A deployment is just like turning the key when starting a car. Only because the engine starts, it doesn’t guarantee that you can actually drive it. Integration tests should be a part of the continuous feedback cycle. As soon as a part doesn’t fit with its surroundings, you should be notified as soon as possible, not four weeks later.

Integration testing is a broad topic and there are different types of integration tests, which needs be handled differently when incorporated in a deployment pipeline. But that’s a subject for a later blog post. All in all, integration tests are important. If you wan’t to have a streamlined development process, where getting new features into production fast without sacrificing quality, you’ll need to have proper integration testing in place.

Tommy Tynjä
@tommysdk

Jul 13 2012

Time for a new take on enterprise testing

Posted by: TommyTynjä @ 16:06

TDD and test automation tools such as JUnit, Selenium etc. have been around for such a long time, that you would expect every modern application platform to leverage from test automation by now. But still it amazes me how little proper testing is adopted in the enterprise today. Usually, the test frameworks are in place, but the development teams lacks the discipline to keep coverage up during an extended period of time. I have gotten the feeling that some companies have given up on automated testing, like “we tried TDD and unit tests, but it didn’t work out for us”. Maybe the tools weren’t mature enough, or the organization itself. Maybe getting a buggy software out to market fast, rather than securing its quality was more important for stakeholders at the time. Maybe it was an organizational concern where testing belonged to the test department and the test department only. It’s always easy to afterwards say that something should have been done differently. However, I think there are plenty of reasons to revisit the thought on proper automated enterprise testing. We are half way through 2012 and there are so many amazing test tools and frameworks available that every opportunity exist to make complete and automated test suites, from unit tests to integration tests and acceptance tests.

Unit tests are obviously the most fundamental part. I’ve been writing unit tests so much, that I now have difficulties writing code without tests! For instance, the other day I was reading a book on a for me new JVM language. The book started out with covering basic operations in the language, and I was encouraged to try out the example code in a REPL. One of my first thoughts though was, “alright, but how do I write an equivalent unit test”! Obviously, a REPL is perfect for trying out language features, but in an enterprise context, it is just as natural to try out new code through unit tests. For me, it’s natural that the first context which uses new code is a test context, and I believe it should be for every developer. Testing isn’t something that is separated from development, it’s part of the development process. There are so many reasons why you should have automated testing, that the excuse “we didn’t have time” just don’t cut it.

Hardly no-one disagrees with the fundementals of testing and why you should do it, so lets focus on how to take enterprise testing to the next level instead. I plan to share more of my thoughts on automated enterprise testing in future blog posts, what lies beyond unit tests, what you should test and how you can do it. So keep a close eye on this space!

Tommy Tynjä
@tommysdk

Jun 08 2012

Generate Java code from an existing WSDL with Apache CXF

Posted by: TommyTynjä @ 09:55

When developing web services, there are occasions when you would like to generate Java code from an existing WSDL document. You might adopt the contract-first approach or just have an existing web service running on a non-Java platform which you want to migrate to Java. You can then start implementing your service from the generated classes. You can use the Apache CXF wsdl2java tool to generate fully annotated JAX-WS (Java EE 6) compliant Java code. Generating the Java code is as simple as:

apache-cxf-2.6.1\bin$ wsdl2java -p se.diabol.api.ws -d /home/tommy/ /home/tommy/webservice.wsdl

… where the -p parameter sets the target package namespace to se.diabol.api.ws, -d points to the target directory for the generated files and the last parameter specifies the source WSDL path.

Resources:
Apache CXF home
Apache CXF wsdl2java reference

Tommy Tynjä
@tommysdk

Jun 04 2012

Add Maven dependencies to your Arquillian micro-deployments

Posted by: TommyTynjä @ 23:55

Arquillian is a testing framework which lets you write real integration tests, run inside the container of your choice. With Arquillian you will be writing micro-deployments for your tests, small Java artifacts that is, which contains the bare minimum of classes and resoruces needed for your test to be executed within your container. To build these artifacts you will be using the ShrinkWrap API. You don’t have to adopt the micro-deployment strategy of course, but do you really want your whole project classpath available for the test of e.g. a single EJB component? Micro-deployments will isolate your test scenario and deploy to the container much faster than if you would bring in the entire classpath of your project.

When writing integration tests with Arquillian and ShrinkWrap you will probably, sooner or later, run into a use case where your test depend on a third party library. Since it would be very inconvenient to declare all the necessary classes in the third party library by hand, ShrinkWrap provides a way to attach complete libraries to your micro-deployment. If you’re using Maven, the Maven dependency resolver feature is very convenient. The 1.0.0.Final version of Arquillian is using a 1.0 beta version of the ShrinkWrap resolver module. The resolver API has in my opinion improved a lot for the latest 2.0 version (currently in Alpha) and I really recommend anyone using the resolver API to use the 2.0 version instead.

If you want to use the latest resovler API, you have to add it to the dependencyManagement tag in your Maven pom.xml before the actual Arquillian BOM, to make sure the 2.0 version of the resolver module will be loaded first:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.jboss.shrinkwrap.resolver</groupId>
            <artifactId>shrinkwrap-resolver-bom</artifactId>
            <version>2.0.0-alpha-1</version>
            <scope>test</scope>
            <type>pom</type>
        </dependency>
        <dependency>
            <groupId>org.jboss.arquillian</groupId>
            <artifactId>arquillian-bom</artifactId>
            <version>1.0.0.Final</version>
            <scope>test</scope>
            <type>pom</type>
        </dependency>
    </dependencies>
</dependencyManagement>

To add third party libraries from your Maven dependencies to your micro-deployment, use the ShrinkWrap resolver API to add the necessary libaries to your artifact. In the example below, the commons-io and json libraries are specified with their respective Maven coordinates:

@Deployment
public static Archive createDeployment() {
    return ShrinkWrap.create(WebArchive.class, "fileviewer.war")
            .addClass(EnableRest.class)
            .addClass(FileViewer.class)
            .addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml")
            .addAsLibraries(
                DependencyResolvers.use(MavenDependencyResolver.class)
                    .artifact("commons-io:commons-io:2.1")
                    .artifact("org.json:json:20090211")
                    .resolveAsFiles());
}

As you can see, it’s very easy to add libraries from your Maven dependencies to your ShrinkWrap micro-deployments. The behaviour of the ShrinkWrap resolver API can also be customized far more than what was shown in the above example. It is worth noting that in the example, any dependencies of the specified artifacts will be included as well. Transitivity is however one aspect which can be customized further through the API. An example project which uses the ShrinkWrap Maven dependency resolver can be found on GitHub.


Tommy Tynjä
@tommysdk

Tags: , ,

May 31 2012

Bind a string to JNDI in JBoss 7.1

Posted by: TommyTynjä @ 16:11

In JBoss 7.0.x there was no convenient way to bind an arbitrary java.lang.String to JNDI. This is however possible in JBoss 7.1.x. Just add your binding to the naming subsystem in standalone.xml such as:

<subsystem xmlns="urn:jboss:domain:naming:1.1">
    <bindings>
        <simple name="java:global/3rdparty/connection-url" value="http://localhost:12345"/>
    </bindings>
</subsystem>

You can then inject the value of your binding into your Java code through:

@Resource(mappedName ="java:global/3rdparty/connection-url")
private String serviceRemoteAddress = null;

To verify your JNDI bindings in JBoss 7.1.1 you can use the jboss-cli located in your $JBOSS_HOME/bin. Start the jboss-cli and type “/subsystem=naming:jndi-view”, such as:

[standalone@localhost:9999 /] /subsystem=naming:jndi-view
{
    "outcome" => "success",
    "result" => {
        "java: contexts" => {
            "java:" => {
                "TransactionManager" => {
                    "class-name" => "com.arjuna.ats.jbossatx.jta.TransactionManagerDelegate",
                    "value" => "com.arjuna.ats.jbossatx.jta.TransactionManagerDelegate@68fc12"
                },
...
Tommy Tynjä
@tommysdk

Tags:

May 24 2012

Continuous Delivery in the cloud

Posted by: @ 20:05

During the last year or so, I have been implementing Continuous Delivery and delivery pipelines for several different companies. From 100+ employees down to less than 5. And in several different businesses. It is striking to see that the problems, once you dig them out, are soo similar in all companies. Even more striking maybe is that the solutions to the problems are soo similar. Implementing a low cycle time – high thoughput delivery pipeline in IT departments is very hard, but the challenges are very similar in most of the customers I have worked with. Implementing the somewhat immature tools and automation solutions that need to be connected and in sync (GO, Jenkins, Nolio, Rundeck, Puppet, Chef, Maven, Gails, Redmine, Liquibase, Selenium, Twist.. etc) is something that needs to be done over and over again.

This observation has given us the idea that continuous delivery is perfectly suited for a cloud offering. Therefore me and my colleges have started an ambitious initiative to implement a complete “from-requirement-to-production-to-validation” delivery pipeline as a could service. We will do it using our collective experience from the different solutions we have done for our various customers.

Why is it that every IT department need to be, not only experts of their business, but also experts in the craft of systems development. This is really strange, in particular since the challenges seem to be so similar in every single IT department. Systems development should be all about implementing the business requirements as efficient and correct as possible. Not so much about discussing how testing should be increased to maintain quality, or implementing solutions for deploying war-files to JBoss. That “development” is done in each and every company these days. Talk about breaking the DRY rule.

The solution to the problem above has traditionally been to try to purchase “off-the-shelf” systems that, at least on paper, fulfills the business needs. The larger the “off-the-shelf” system, the more business needs are fulfilled. But countless are the businesses that have come to realize that fulfilling on paper is not the same as fulfilling in production, with customers, making money. And worse, once the “off-the-shelf” system has been incorporated into every single corner of the company, it becomes very hard to adapt to changing markets and growing business requirements. And markets change rapidly these days – if in doubt, ask Nokia.

A continuous delivery pipeline solution might seem like a trivial thing when talking about large markets like the one Nokia is acting in. But there are many indicators that huge improvements can be achieved by relatively small means that are related to IT. Let me mention two strong indicators:

  1. Most of the really large new companies in the world, all have leaders with a heavy legacy in hardcore systems development/programming. (Oracle, Google, Apple, Flickr, Facebook, Spotify, Microsoft to mention a few)
  2. A relatively new concept “Lean-Startup”, by Eric Ries takes an extremely IT-close approach to business development as a whole. This concept has renderd huge impact in fortune 500 companies around the word. And Eric Ries has busy days coaching and talking to companies that are struggling to be innovative when the market they are in are moving faster than they can handle. As an example, one of the large european banks recently put their new internet bank system “on ice” simply because they realized that it was going to be out of date before it was launched.

All in all. To coupe with the extreme pace, systems development is becoming more and more industry like. And traditional industries have spent more that 80 years optimizing production lines. No one would ever get the stupid idea to try to build a factory to develop a toy or a new component for a car without seeking professional guidance and assistance. I mean, what kind of robot from Motoman Robotics would you choose? That is simply not core business value!

More on our continuous delivery pipeline in the cloud solution will follow on this channel!

Tags: ,

Next Page »