Obtaining the complete XML message as xs:string with XSLT

At a recent project I needed to implement functionality which would store an incoming message in a relational database. Since we were using the WSO2 platform I created a WSO2 DataService for this by generating a Data WebService based in my MySQL table. Each message supplied to this service in the following XML format would be inserted in a MySQL database.

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="originalMessage" type="originalMessageType"/>
  <xs:complexType name="originalMessageType">
    <xs:sequence>
      <xs:element type="xs:string" name="id"/>
      <xs:element type="xs:string" name="content"/>
    </xs:sequence>
  </xs:complexType>
</xs:schema>

Running Hive jobs on AWS EMR

In a previous post I showed how to run a simple job using AWS Elastic MapReduce (EMR). In this example we continue to make use of EMR but now to run a Hive job. Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems.
To create the job in EMR I will still make use of the CLI (written in Ruby) supplied with EMR (for installation see here). The job that I am going to create is described in more detail in the ‘Getting started Guide Analyzing Big Data with AWS’.

Archiving your S3 content

In a previous post I showed how to setup a Glacier Vault and put data in it by using a CLI. Another way to make use of the cheaper storage service is to move your data from a S3 bucket into a Glacier vault. You can do this by making use of the Object Lifecycle Management which make it possible to move data from your S3 bucket to a Glacier storage class by simply applying a rule like if data in a bucket is more than 3 months old move it.

Archiving data to AWS Glacier

For archiving your data at AWS you can use AWS Glacier. This service offers cheaper storage than the S3 service but the downside is that your data won’t be accessible right away as it is with S3. It may take a few hours to restore the data but if you are using this service for real archiving purposes this shouldn’t be a real issue. Lets setup a Glacier Vault by using the Management Console. Select the Glacier service in the Management Console:

Creating Mock service in WSO2 ESB

I posted before about how to create a mock service when you have a JDK 1.6 or higher available (for the interested you can find it here). Of course there are a lot more options to mock web services (one I used often is the option supplied by SoapUI). This post shows a way to mock a web service by using a WSO2 Proxy Service.
The way to do this is by using the PayloadFactory mediator. There is also an example of its general usage here

Using AWS ElasticMapReduce with the Command Line Interface

In this post I am going to use the AWS MapReduce service (called ElasticMapReduce) by making use of the CLI for EMR.

The process of using EMR can be divided in three steps on a high level:

• set up and fill the S3 buckets
• create and run a EMR job
• get the results from the S3 bucket

Before you can start with these three high level steps there are some other things that have to be arranged:

• you will need to have an AWS account
• you will need to install an S3 client on your machine
• you will need to install EMR CLI on your machine

Well, for the account for AWS I simply assume it is there otherwise it is about time to get yourself one

As an S3 client I make use of s3cmd of which I described the installation here.

Using s3cmd for working with AWS s3 buckets

When you make use of (some of the) Amazon services there is a big chance S3 is one of them for the storage of files, be it temporary or more permanent. When you are using this service of AWS and you want to combine this with scripting possibilities you should have a look at the s3cmd library.
I installed the tool on my Mac and use it quite often (automatic backups of certain directories for instance). It makes working with S3 buckets in the Terminal easy. The installation is straightforward although for a Mac you might need to install a PGP tool. For a step by step installation see this great post. After finishing this you are ready to play with AWS S3 from your Mac.

AWS Summit London 2013

Last week I attended to the AWS Summit in London. Together with a few colleagues from 4Synergy we saw a lot of interesting stuff.
I attended to the Data & Analytics Track and saw (again) the potential of what AWS can mean to businesses that have to deal with Big Data or extremely busy peaks in their web traffic. If you are interested in the presentations held that day you can find them here.
If you want to know what AWS can mean for your business just contact us. As AWS Solution partner we are always keen to discuss the possibilities of AWS (or Cloud technology in general) with you.

Implementing WireTap EIP in WSO2 ESB Proxy Service

One of the EIP patterns is the WireTap. In this pattern you can extract a copy of the message from the flow without interfering the process.

Wire Tap

The way to do this in the WSO2 ESB is by simply using the Clone mediator in the proxy service.

Deploying the WSO2 CAR file with Maven

To deploy my WSO2 Carbon Application aRchive (CAR) file to my WSO2 ESB instance I use this Maven plugin. I couldn’t find a lot of documentation about it but I post here how I use it. I added the following plugin to my pom.xml file:

<plugin>
  <groupId>org.wso2.maven</groupId>
  <artifactId>maven-car-deploy-plugin</artifactId>
  <version>1.0.0</version>
  <extensions>true</extensions>
  <configuration>
    <carbonServers>
      <CarbonServer>
        <trustStorePath>${basedir}/src/main/resources/security/wso2-localhost.jks</trustStorePath>
        <trustStorePassword>wso2</trustStorePassword>
        <trustStoreType>JKS</trustStoreType>
        <serverUrl>https://localhost:9443</serverUrl>
        <userName>admin</userName>
        <password>admin</password>
        <operation>deploy</operation>
      </CarbonServer>
    </carbonServers>
  </configuration>
</plugin>

I think this plugin is pretty straightforward. As said before I couldn’t find any documentation about it.
The last thing to do to make this work is to create the trustStore file. There are two ways (at least) to get it. The easy way is to get it from the installation directory of the WSO2 ESB. The file is named ‘client-truststore.jks’ and you can find it in the directory ‘$CARBON_HOME/repository/resources/security’. The password is ‘wso2carbon’.