Running MapReduce Design Patterns on Cloudera’s CDH5

9 Sep

cloudera-hadoopOne of the better books I read so far about MapReduce is ‘MapReduce Design Patterns‘ as I mentioned in my previous post. In this post I describe the steps to get started with running the Hadoop source code that goes with the book on Cloudera’s latest Hadoop distribution CDH5. I decided to be making use of HDFS and YARN for testing the patterns. Take the following steps to get it all up and running:

  • Get CDH5 and run it
  • Install IntelliJ IDEA
  • Upgrade GIT client
  • Create local directory
  • Checkout source code
  • Install source data
  • Run the job

Continue reading

Hadoop and MapReduce Design Patterns

14 Aug

hadoopRecently I finished my last project in which I was implementing Mule ESB. This gives me some room in my schedule to dive into the world of Big Data again (more specifically the Hadoop ecosystem). I have looked into this subject before which resulted into several blog posts. This time I started with a refresh by taking the online training of AWS: Big Data Technology Fundamentals. It is about MapReduce, Hadoop, Pig and Hive. After this nice online training I started with the Hadoop training of core-servlets. I had to get used to the form and layout of the training but now I have been working with it for a while I realise it contains a lot of information about the way Hadoop works. It comes with a (working!) virtual machine (based on Cloudera’s CDH4) on which Hadoop and the necessary tooling is installed including all training and exercise materials (and solutions).
Paralel to this (low level) training I am going through the book MapReduce Design Patterns. With this book you get a good idea which problems you can manage/solve with MapReduce framework and in what way. Especially the recommendation when not to use a certain pattern can be very handy while working with MapReduce.

Mule ESB, ActiveMQ and the DLQ

27 Jul

Apache-activemq-logoIn this post I show a simple Mule ESB flow to see the DLQ feature of Active MQ in action.
I assume you have a running Apache ActiveMQ instance available (if not you can download a version here). In this example I make use of Mule ESB 3.4.2 and ActiveMQ 5.9.0. We can create a simple Mule project based on the following pom file: Continue reading

Up And Running With Spring Framework 4 In Five Hours

29 Jun

spring-courseI recently was invited to follow an online training at Udemy to get up to speed with Spring 4. Now most of the time I am not really interested in the subject of the offered trainings but as an open source Java developer you sooner or later will run into the Spring Framework. In my case I have been dealing a lot with the Mule ESB the last few years which is based on the Spring Framework. So deciding it wouldn’t harm to gather some new information about the Spring Framework I decided to give it a try.
I have been following up to 70% of the course now and so far I like ir. The course has found the right balance between going into depth about some of the subjects or explain them globally and leave it up to the trainee to find out more about it in detail. Also using video’s of approx. 10 minutes makes it easy to watch a few of them one night and leave the rest for later.
So if you are working with an older version of Spring or you want an introduction to the framework you should give this training a try. If you use this link this link you will receive over a 70% reduction so you can get access to the training for 39 USD 21 USD (only valid till 5th of July!). You can also see a few free video’s to see if it will fit your need.

After completing the course you even receive a certificate :-)

Enabling WSSE Basic Authentication for a Web Service client in Mule

22 Jun

In a previous post I described how to enable WSSE security options in your web services proxied by Mule ESB. In this post I show you how to enable WSSE basic authentication on a ‘client’ side. I used this set up when I needed to consume services from Oracle CRM On Demand. Continue reading

Running the Maven Release Plugin with Jenkins

12 May

Recently I started to implement our release process in Jenkins. Until then I just ran the release plugin of Maven on my local machine which did the job. As I said we decided to move this task to Jenkins. The build/release toolstack was:

To show my Jenkins configuration I have setup a very basic Maven module named ‘myapp’ with the following pom: Continue reading

Unexpected behavior using Mule Expression Language

6 Apr

mulesoft-logoWhile developing flows with a recent Mule ESB there is a big chance you will make use of MEL in your configuration. Although this feature has added great benefits while developing Mule flows it sometimes drives me crazy. In this post I will show two examples which took me some time to get it working.
The first was when I was using an expression-transformer to get a part of an XML document as payload. The expression I tried first was: Continue reading

Using encrypted passwords with Mule ESB

9 Mar

A good practice in the Mule ESB is to supply properties by using a property file. Most of the time you will end up adding passwords to the properties file. In that case you might want to encrypt the passwords so it is not visible for every one who has access to the property file. Mulesoft described how to do this in combination with the Mule ESB. Although it is a good starting point I thought it may help to create a complete example so I put all the steps in this post. There are two environments that have to be modified: the development environment and the runtime environment.
Continue reading

Unit testing with Mule ESB 3.4 Enterprise Edition

12 Feb

There is a big chance that when you are using the Mule ESB EE as your ESB implementation and you are creating functional/unit tests you sooner or later will need a license on your test classpath to be able to run Mule in your unit tests. Luckily this isn’t very hard to accomplish. You can just take the license file that is generated when installing your license at the (standalone) server. You can find the license file on the server here: $MULE_HOME/conf/muleLicense.lic. Just copy it to your
‘$PROJECT_HOME/src/test/resources/’ folder so it an be accessed on the classpath.
Till so far fo the happy scenario. However after doing this I ran into the following exception

Caused by: java.lang.ClassNotFoundException: com.mulesource.licm.LicenseKeyException

Continue reading

Message Throttling with Mule ESB

31 Jan

mulesoft-logoWhile implementing the Mule ESB I ran into the following scenario:
Messages were picked up from a queue, the message was transformed, offered to a web service, the response was transformed and put on a response queue.
Not an unusual use case in the world of integration. One thing that made this one special was the (lack of) performance of the web service. It appeared as soon as we started to send messages to the services via the ESB the response time for each of the messages increased significally. Investigation showed that the web service was having serious issues when concurrent messages were sent. This is what the Mule ESB do by default. It will start of with a set of threads to process the messages as quickly as possible. Until the issue with the web service is solved I decided to add throttling functionality to my flow so I could manage the number of calls to the web service. Continue reading


Get every new post delivered to your Inbox.

Join 101 other followers