At a recent project I needed to implement functionality which would store an incoming message in a relational database. Since we were using the WSO2 platform I created a WSO2 DataService for this by generating a Data WebService based in my MySQL table. Each message supplied to this service in the following XML format would be inserted in a MySQL database.
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="originalMessage" type="originalMessageType"/>
<xs:element type="xs:string" name="id"/>
<xs:element type="xs:string" name="content"/>
In a previous post I showed how to run a simple job using AWS Elastic MapReduce (EMR). In this example we continue to make use of EMR but now to run a Hive job. Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems.
To create the job in EMR I will still make use of the CLI (written in Ruby) supplied with EMR (for installation see here). The job that I am going to create is described in more detail in the ‘Getting started Guide Analyzing Big Data with AWS’. Continue reading
In a previous post I showed how to setup a Glacier Vault and put data in it by using a CLI. Another way to make use of the cheaper storage service is to move your data from a S3 bucket into a Glacier vault. You can do this by making use of the Object Lifecycle Management which make it possible to move data from your S3 bucket to a Glacier storage class by simply applying a rule like if data in a bucket is more than 3 months old move it.
For archiving your data at AWS you can use AWS Glacier. This service offers cheaper storage than the S3 service but the downside is that your data won’t be accessible right away as it is with S3. It may take a few hours to restore the data but if you are using this service for real archiving purposes this shouldn’t be a real issue. Lets setup a Glacier Vault by using the Management Console. Select the Glacier service in the Management Console:
I posted before about how to create a mock service when you have a JDK 1.6 or higher available (for the interested you can find it here). Of course there are a lot more options to mock web services (one I used often is the option supplied by SoapUI). This post shows a way to mock a web service by using a WSO2 Proxy Service.
The way to do this is by using the PayloadFactory mediator. There is also an example of its general usage here Continue reading
In this post I am going to use the AWS MapReduce service (called ElasticMapReduce) by making use of the CLI for EMR.
The process of using EMR can be divided in three steps on a high level:
- set up and fill the S3 buckets
- create and run a EMR job
- get the results from the S3 bucket
Before you can start with these three high level steps there are some other things that have to be arranged:
- you will need to have an AWS account
- you will need to install an S3 client on your machine
- you will need to install EMR CLI on your machine
Well, for the account for AWS I simply assume it is there otherwise it is about time to get yourself one
As an S3 client I make use of s3cmd of which I described the installation here.
When you make use of (some of the) Amazon services there is a big chance S3 is one of them for the storage of files, be it temporary or more permanent. When you are using this service of AWS and you want to combine this with scripting possibilities you should have a look at the s3cmd library.
I installed the tool on my Mac and use it quite often (automatic backups of certain directories for instance). It makes working with S3 buckets in the Terminal easy. The installation is straightforward although for a Mac you might need to install a PGP tool. For a step by step installation see this great post. After finishing this you are ready to play with AWS S3 from your Mac.
Last week I attended to the AWS Summit in London. Together with a few colleagues from 4Synergy we saw a lot of interesting stuff.
I attended to the Data & Analytics Track and saw (again) the potential of what AWS can mean to businesses that have to deal with Big Data or extremely busy peaks in their web traffic. If you are interested in the presentations held that day you can find them here.
If you want to know what AWS can mean for your business just contact us. As AWS Solution partner we are always keen to discuss the possibilities of AWS (or Cloud technology in general) with you.
One of the EIP patterns is the WireTap. In this pattern you can extract a copy of the message from the flow without interfering the process.
The way to do this in the WSO2 ESB is by simply using the Clone mediator
in the proxy service. Continue reading
To deploy my WSO2 Carbon Application aRchive (CAR) file to my WSO2 ESB instance I use this Maven plugin. I couldn’t find a lot of documentation about it but I post here how I use it. I added the following plugin to my pom.xml file:
I think this plugin is pretty straightforward. As said before I couldn’t find any documentation about it.
The last thing to do to make this work is to create the trustStore file. There are two ways (at least) to get it. The easy way is to get it from the installation directory of the WSO2 ESB. The file is named ‘client-truststore.jks’ and you can find it in the directory ‘$CARBON_HOME/repository/resources/security’. The password is ‘wso2carbon’. Continue reading