Getting started with Python on Spark

At my current project I work a lot with Apache Spark and running PySpark jobs on it. For those who also want to get their hands dirty with Spark in combination with Python I can recommend this course at Udemy. It gives a broad introduction about Spark in general, the different modules like ‘Spark Streaming‘, ‘Spark Sql‘, ‘MLlib‘ and ‘GraphX‘, and how to use Python to make use of the Spark system.
It also explains how you can run your own cluster in the cloud with AWS EMR about which I wrote several post before. And no worries, after completing this course there is still lots more to discover about Spark 😉 but like I said this should be sufficient to get started.

Small hack to avoid SSL validation in Spring RestTemplate

As you might notice from my previous posts I am a big fan of Spring Framework. A few weeks back, for example, I needed to set up a REST API client to be able to test some things in our Development environment. By using Spring Boot and RestTemplate I was able to create this quickly. Unfortunately I ran into the issue that the API endpoint was using a SSL certificate that wasn’t supplied by a trusted ‘default’ CA. So in this case I wanted to switch this SSL validation of in the RestTemplate.

AWS DevOps Engineer- Professional: Got it!

When you have a look at the archives of my blog you might notice that I haven’t written any blogs for the last few months. The reason was simple: lack of time. I had decided to not only renew my existing AWS certificate but take that to the next (and highest) level, DevOps Professional, and also get the more generic Solution Architect – Associate one. Two months ago I got the Solutions Architect one which was actually not that difficult. But to get the Professional level took a bit more preparation. Nevertheless last Friday I successfully took the (almost 3 hours taking) exam and got the certificate:

So with this certificate added to my list my collection of badges look like this:

• DevOps Professional Badge
• Architect Associate badge
• Developer Associate badge

If you want to validate an AWS certificate you can use the following link: verification. My validation number is 24KQF0WKKMF4Q63B if you want to test it 🙂

Use Spring Cloud Config as externalized configuration

One of the most important practices when it comes to continuous delivery is building your binaries only once and use that one in the different environments (dev, test, acceptance, etc). This means that most likely you will need to externalize your configuration of your application. With Spring Boot applications it comes down to keep the values of your properties in your application.properties outside of your application jar. By doing this you can use the same jar for deployments on different environments (An alternative is to package the values of the properties for all possible environments with your jar but I don’t consider that as a good practice in most situations).
Especially when you are building and deploying Spring Boot applications you should have a look at the Spring Cloud Config project. In this post I will go through the steps as described here and you will see how to setup your own configuration service in just a few steps.

Pipeline as code with a Spring Boot application

This is the last in a serie of posts about continuous delivery based on my local Docker compose stack (see the first and second post here). In this post I use a simple Spring Boot project to show how to make use of the ‘pipeline as code‘ concept. Please note that this is only an example and much, much more is possible. The application I use is taken from the Spring Boot site. The Jenkinsfile is inspired by the one in this post but I had to modify some things to have it work with my stack. The sources of my project can be found here. I will explain the most important snippets in this post.

Configure Jenkins for Continuous Delivery of a Spring Boot application

In my previous post I described how I started a continuous delivery stack with one simple command. The next step is to prepare the stack to build and deploy an application in an automated way. In this post I describe how to configure the stack so it is ready for processing a simple Spring Boot application.
Although I have the components running as Docker containers they still need configuration to have them work together, especially Jenkins.

Set up your own Continuous Delivery stack

Last week I wanted to try new things with ‘pipeline as code’ with Jenkins. The best way to try new things is running it as Docker containers. This way I can keep my MacBook clean and don’t mess up existing stuff I am working on (also see this article about what Docker can offer for a developer). Another big advantage by using Docker is that there is already a complete stack of Dockers available to set up the necessary tools. However, for some unclear reason this stack didn’t work on my MacBook (see this issue) so I took this opportunity to build my own stack with some Dockers of my own choice :-).
The tools in my stack are:

• Nexus (as Maven repository)
• Gitlab (to host the sources of my Java projects)
• Jenkins (of course to make use pipeline as code)

Using NAT Gateway instead of NAT Instance with AWS VPC

I recently bumped into my own post of some time ago in which I describe how to add a NAT instance to your private subnet to have access to the internet to install packages etc. Although this still works some time ago AWS introduced the NAT Gateway which in most cases makes life much easier.
In this post I show you how to set up the NAT Gateway instead of using the NAT instance. To get the same start situation as the original post I created this CloudFormation script that creates a VPC with two private and public subnets. When these are in place I can create an EC2 instance in both the private subnet and in the public one, as I also described in the original post.

Hip Hip Hooray, 10 years of blogging

This month it has been 10 years ago that I published my first blog post. The post was about Oracle BPEL which was quite hot back then. Of course lots has changed in IT since then but not my eagerness to learn new things and writing about them.
Last week I received a mail that my blog was selected as one of the top 40 Java blogs on the web (not sure if that is good as I was in the top 5 in 2015 :-/). But nevertheless I guess this is one of the reasons I am writing these posts: to help out others as other blogs are often helping me.
Another important reason is getting new assignments by showing others what I am working on. Besides starting to blog ten years ago I also started working as a IT freelancer. This blog brought me several freelance assignments of which the nicest one was a three week job in Mauritius during the Dutch winter season 😎
So I intend to keep on posting for at least the next ten years. There is still enough to learn as I notice every day and who knows what it might bring.

Making Spring Boot application run serverless with AWS

In several previous posts I described how to setup your Spring Boot application and run it on AWS Elastic Beanstalk. Although this is a great step to go from a physical server to one in the cloud there is an even better step possible! Going serverless. That means no costs for any server and no maintenance or configuring of servers! That sounds good, right? AWS has made it quite easy to go serverless with the combination of AWS Lambda and AWS API Gateway. In this post I will describe what it took for my Spring Boot application that runs on Elastic BeanStalk to run the same functionality serverless.