CALL +44 (0)20 7183 3893
Blog

Friday, 3 May 2013

Whisky Web - Not a normal developer conference

I recently attended Whisky Web 2 - the second annual Whisky Web conference for web developers, held in Scotland. The second installment of Whisky Web was held on the 12th-13th of April at Airth Castle in Stirlingshire and in Edinburgh.

As a first-time conference-goer, I am sharing my experience of this new and exciting conference. The fact that I am happily writing this post in vim for a challenge, having never gone anywhere near command-line editors before, is a testament to the adventurous spirit that pervades the Whisky Web conference and all those involved in it.

In the middle of the week before Whisky Web, I was chatting in the pub with my co-worker Max Manders who is one of the organisers of the event. He was clearly very passionate about the conference that he has helped to create and excited to have brought our company on-board as a sponsor, alongside other big names such as Facebook and Adobe. I was keen to know more about the conference and was highly intrigued when I heard the words "whisky" and "castle". At that point I did not know that I would actually be attending Whisky Web.

The next day, back at work, I received an email from a co-worker giving me the news that my name had come out of a hat in a prize-draw. I had won a free ticket to Whisky Web! The free ticket had been awarded to Cloudreach in return for Cloudreach's sponsorship of the conference, and I had put my name into a hat the week before simply by replying to an email. I was extremely pleased with my good fortune and determined to make the most of the opportunity.

Friday 12th April, Day One

The opening keynote speech to Whisky Web was scheduled for 9am outside the Parliament Building in Edinburgh. After the keynote, we would travel by coach to Airth Castle.

The two most notable things about the keynote were that it was held outside (a thumbs up from me!), and also that it was within the shadow of Arthur's Seat - Edinburgh’s spectacular natural landmark. You really could not fail to miss the meeting point, even if you had already failed to notice the high concentration of geeks with their long hair and rucksacks.

I don't have long hair, but I certainly had my rucksack on as I strolled briskly down the ringroad around Arthur's Seat, taking in the sights of Holyrood park. Although it was already mid-April, Edinburgh seemed not to have noticed yet and it was a wintery and dull morning. Probably not the best day to hold a keynote speech outside.

Especially when the keynote speaker had cancelled! In the end that did not matter as the minor let-down was quickly forgotten during the creatively-improvised speech and the rest of the weekend. We were glad however once we were onboard the coach and heading towards Airth Castle.

I had already made friends with an easy-going German developer with long hair. I was already learning a lot about the development community, life in Germany and also myself as we exchanged experiences.

We arrived at Airth Castle and checked in to the hotel/spa where the conference talks would be held and where we would drink and sleep. The hotel was a blocky modern building which was located in the grounds of Airth Castle.

After a minor mis-adventure looking for my room, I was directed towards the castle itself and then it dawned on me that along with my free ticket i’d lucked into having a luxury suite inside Airth Castle! I immediately got completely lost in the unpredictable winding staircases and curved hallways. In fact I got lost every time I went inside that castle.

About two hours later I heard two words that would change my life as a developer: "vim adventures".

As a developer or regular unix user you roughly know what vim is. You even have to use it sometimes, but mostly that is because an application like git (exactly) has dropped you into vim in edit mode as its default editor of choice. Thus I had only ever learned one command in vim up to that point::q! (how to quit).

Vim Adventures is an adventure game that teaches you how to use vim by using vim commands as its controls. My introduction to it occurred early on in my second talk of the day, given by Rowan Merewood - who is a charismatic and fun-loving speaker with a fresh perspective on how we can be better developers. If you like puzzle games of the ilk of (to paraphrase Merewood) “The Legend of Zelda” and also a mild interest in what vim is and how to use it, you should visit http://vim-adventures.com without a moment's thought.

Rowan’s talk fired me up for the rest of Whisky Web and I was excited to realise that it would not be all jargon-filled, deeply-technical lectures and that I could expect to have a lot of fun.

Throughout the weekend I was repeatedly reminded of the idea that Whisky Web is as much about having fun and meeting like-minded people as learning cool new things. That evening we had the whisky tasting session, held in a function room on the ground floor of Airth Castle itself.

After freshening up, I left my hotel room and strolled down a winding staircase into the throng. As a big fan of whisky I was pleased to find myself being given advice by whisky experts on how to taste whisky and such. We were given some background information on the whisky we were tasting, and then they brought out the 18-year old single-malt Dewar’s whisky which blew us all away with its intensity. The whisky-tasting was hugely popular.

We enjoyed a brilliant five-course meal back in the hotel, at tables laid with cutlery all over the place. I found myself at a table of good chat and everybody had a great evening, until somebody flew a flying shark into the room and immediately got it stuck in the rafters of the building way above us! - How many drunk developers does it take to get a flying shark back down … ?

One of the things that characterised Whisky Web was the very high diversity of nationalities amongst the attendees - surprising considering this was only the second Whisky Web event and it reflects how much of the spirit of the community that Whisky Web gives off.

Saturday 13th April, Day Two

We left the fine environs of Airth Castle and jumped back in the coach for a trip back to Edinburgh. Amazingly, Whisky Web had booked out the Surgeon's Hall Museum in Edinburgh for the day's events.

We were given free reign to walk about and explore inbetween talks, of which there were eight in total - two at a time - so that we still had four hours of talks to listen to.

The talks on the second day were as fun and as varied as those on the first. From each talk I took away at least one tangible piece of information about a new tool or technique that I would later investigate or implement. In fact I still couldn't wait to get home on Sunday to start playing Vim Adventures!

After all the talks and the closing keynote, the evening was spent partying in the private upstairs room of the Ghillie Dhu pub in Edinburgh - a church that has been converted into a pub. A lot of free drinking was involved and we enjoyed a lively disco and live music. In all, a good way to round off Whisky Web. I felt that the organisers of Whisky Web had done a great job with their highly creative choice of venues for the event.

In Conclusion

I took away a lot from Whisky Web - from the talks, the many lively conversations and the people that I met. I discovered amazing new tools, new ways to think and do things and rekindled my excitement for the creativity and unique challenge of software development.

It was pointed out during the opening keynote that one of the principles of Whisky Web is to keep the number of attendees to a medium size (around 100), so that it would be possible to meet nearly everybody at the event and to feel a sense of shared experience. By the end of Whisy Web I felt that this nice idea had been well-borne out in reality.

Having never been to a conference before Whisky Web, I am now much more open-minded towards the concept. Even from a purely agnostic point of view, I can highly recommend Whisky Web to any developer as a worthwhile way to spend a couple of days and gain a fresh perspective. I absolutely intend to be a paying attendee to future installments!

Wednesday, 20 February 2013

CloudStack vs OpenStack

Kickin’ in the front seat
Sittin’ in the back seat
Gotta make my mind up
Which seat can I take? 
Rebecca Black, Friday

Two years ago, comparing OpenStack and CloudStack, Robert Paulson wrote that: “Cloud computing is very much like sex in high school. Everyone's talking about it and few people are actually doing it”.

It is 2013 and things have changed. Cloud computing moved from high school to Uni. Everyone’s doing it, but only few people are doing it in the right way (Cloudreach is amongst those few, of course).

Many fine articles can be found on the Internet which compare CloudStack, Openstack, and other private/hybrid cloud platforms. Most of them just go through feature lists from OpenStack and CloudStack websites, mention that there is more buzz around OpenStack, and conclude that both platforms are great and it is up to you to decide. So at the end of the day, techie guys still have to spend valuable time installing and evaluating both platforms.

This blogpost is an attempt to point out most notable differences between OpenStack and CloudStack which have attracted our attention. We start from obvious things like popularity and size of surrounding communities, move to more pleasurable features like VPNs and conclude that both platforms are fine. As a matter of fact, they are, trust me.

Monday, 14 January 2013

Comparing Amazon VPC connectivity options

In August 2009 Amazon announced its Virtual Private Cloud (VPC) service, essentially giving enterprise customers worried about security and control in the cloud a solution to that concern. Since then the Amazon VPC has matured as more and more services have become available from within the VPC.

Amazon Virtual Private Cloud allows IT administrators to provision a private, isolated section of the Amazon Web Services (AWS) Cloud where they can launch AWS resources in a virtual network that they define. They can have complete control over the virtual networking environment, including selection of IP address ranges, configuration of routing tables, subnets and network gateways.

Furthermore customers can connect their existing data centers and branch offices to the Amazon VPC and access the AWS cloud as if it is an extension of the corporate network. This connectivity between the corporate offices and the Amazon VPC can be accomplished in several ways.

In this short blog, we will explore the options available for connecting the enterprise network to the Amazon VPC whilst we compare and contrast the advantages, disadvantages and associated costs.


Amazon Direct Connect


AWS Direct Connect is an AWS service that allows you establish a dedicated network connection between your WAN network and the Amazon Web Service global network. If your corporate network has presence in one of these locations, Direct Connect facilitates dedicated 1G or 10G connectivity between your network equipment at that location and Amazon's routers.

Pricing information can be found here.

If connecting in London Telecity, a single 1G port will cost at least $223 per month for the port connection-hours. Additionally you pay $0.03 per GB for data transfers outbound from the VPC to the corporate network. Furthermore, if your corporate offices and datacenters are already reachable from the Direct Connect peering location across the enterprise WAN, only minimal configuration will be required to route traffic between the VPC and those offices.

Advantages

  • Reduces bandwidth costs for traffic-heavy applications.
  • Provides consistent network performance compared to other options.
  • Can be used for accessing AWS services outside the VPC.

Disadvantages

  • Requires existing network presence in a very limited set of locations.
  • Requires more complex network hardware and configuration, for example 802.1q VLANs, BGP ..etc.
  • If the traffic loads are not heavy enough, this is an expensive option.
  • Not very elastic, the options are 1G or 10G ports, there is nothing in between. 

Monday, 7 January 2013

Varnish and Autoscaling... a love story


While working on a cool project at Cloudreach, I stumbled upon Varnish, and fell in love with it instantly. The first thing I tried to do was to combine Varnish with the awesomeness provided by AWS Elastic Load Balancer (ELB), in a combination which looks like:





While the frontend ELB works out of the box with Varnish (no surprises here), the backend ELB doesn’t work as expected with Varnish. The problem lies on the fact that Varnish is resolving the name assigned to the ELB, and it’s caching the IP addresses until the VCL get’s reloaded. Because of the dynamic nature of the ELB, the IPs linked to the cname can change at any time, resulting in Varnish routing traffic to an IP which is not linked to the correct ELB anymore.

The problem is discussed here and here but after Googling around I couldn't find any solution which didn’t involve doing:

ELB -> VARNISH -> NGINX (or HAproxy) ->  ELB -> AUTOSCALING GROUP

Going through so many layers seemed too much, taking into consideration that Varnish can be used to load balance requests and perform health checks on the backend nodes without the need for an Internal ELB. The more I thought about it, the more I realised how simple it would be to implement a solution..... so I did it. Using Varnish to perform the load-balancing, removes the overhead of going through an internal ELB, and it will require reloading the backend nodes only when an autoscaling activity takes place.


The solution I've implemented uses varnishadmin command line tool, boto, and some bash scripting to glue all together.

First of all we need to get the backend nodes configured in Varnish and store them on a file:


varnishadm -T $HOSTPORT -S $SECRET backend.list > varnish_ips

Then, we will have to query the autoscaling group, and update the backends if any instance has been added/terminated. The following Python code does most of the job:


Let’s break it down:

  • get_autoscaling_ips gets the IPs associated with instances added to a specific autoscaling group.
  • get_varnish_ips loads the backend IPs in a Python array
  • update_vlc_file compares the two list of IPs. If there is any difference (you might want to reconsider this aspect) in the two lists of IPs, it creates a new VCL file containing the IPs retrieved from the autoscaling group.

In order to decouple the VCL section which is used to define request handling and document caching policies (unlikely to change according to the autoscaling group)  from the section which is used to configure the backends, the Python script outputs the new VCL in the following format:

include /etc/varnish/healthcheck.vcl;

node definitions


director definitions


include /etc/varnish/use.vcl

The node definition and the director definition is dynamically generated by the script, while healthcheck.vcl is a static file where the healthchek conditions are defined (what a surprise:) and use.vcl is another static Varnish config file, which makes use of the director definition.

Once the new VCL is generated, it’s just a matter of reloading it, running:

varnishadm -T $HOSTPORT -S $SECRET vcl.load $NAME $FILE
varnishadm -T $HOSTPORT -S $SECRET vcl.use $NAME


Something I noticed when creating the script, is that backend.list returns the list of the configured backends, regardless if the VCL which defines them is in use or not. This behaviour makes the all exercise of comparing VCL backends with autoscaling IPs useless, so we need to remove all the previous VCL configs running:

varnishadm -T $HOSTPORT -S $SECRET vcl.discard $OLD_VCL

The three scripts can be glued together on a bash script which runs as a cron job on each Varnish server. The code above has not been used in production yet, so please do test thoroughly before usage. II’m always curious to hear of any feedback, so get in touch if you have any comments on this.

As usual, please reach out to us if you need any help or advice using AWS!


Nicola Salvo
System Developer

Tuesday, 23 October 2012

Setting domain wide signatures for google mail

Using Google Apps Script and Google docs

While Google mail’s compliance footers are great for legal compliance, Google currently don’t offer an easy way to set signatures for all your users that are based on user specific variables. I had a quick look at the existing offerings but I also noticed that due to the openness of Google’s APIs it's easy to pull data from a Google spreadsheet and push out the signatures to users across a domain. Giving us:
  • Centrally managed user data in a simple spreadsheet 
  • A legal compliance footer enforced on all outbound emails 
  • A quickly customisable signature with contact details that can be edited/updated by users (no more waiting for IT to update your contact details) 


As Google Apps Script doesn't connect to the Google email settings API directly, this also makes use of Google Apps Script's support of oauth to securely authenticate from Google Apps Script to the Google email settings API. This post includes setup instructions and an overview of how it works, if you just want the code feel free to skip to the end...

Wednesday, 6 June 2012

Auto Scaling and Chef Nodes Deregistration

Creating an infrastructure which scales dynamically according to the traffic volumes hitting a web site, is a reality using Amazon Web Services (AWS). However, there are a number of key design points, which differentiate a professional solution from an ad hoc collection of services.

One of these design principles for auto scaling an application is to strive keep it stateless.

Sometimes this is not possible, for example when OpsCode Chef is used to configure and deploy code to instances created via an auto scaling process. In order to use Chef on a new instance, it needs to first register itself first with a central Chef server and resolve credentials and ssh access so that code and commands can be executed.
The problem with registering autoscaling nodes, is that when those are terminated for one reason or the other, Chef Server is left in a “dirty” status, with references to nodes which are no longer active.

Fortunately, the AWS team have enhanced their platform with lots of additional services complementing EC2, two of which are useful to solve the problem described above.

When an action is triggered in an auto scaling group, this can generate a notification using the Amazon Simple Notification Service (SNS). SNS works across all the services offered by AWS, and an auto scaling group can be configured in order to generate an outbound notification for both scale up and scale down events.
SNS receives and publishes events using “topics”.  A topic is a communication channel to send messages and subscribe to notifications. It provides an access point for publishers and subscribers to communicate with each other.

An SNS topic can be created very easily using the web UI, or using the command line:

sns-create-topic MyTopic

Once a topic for autoscaling events is created, the auto scaling group needs to know where to publish notifications. This can be achieved in one of two ways; either using the command line interface:

as-put-notification-configuration MyGroup --topic-arn arn:placeholder:MyTopic --notification-types autoscaling: autoscaling:EC2_INSTANCE_TERMINATE topic-ARN MyTopic


or using CloudFormation:

"NotificationConfiguration" : {
             "TopicARN" : { "Ref" : "MyTopic" },
             "NotificationTypes" : [ "autoscaling:EC2_INSTANCE_TERMINATE"]
       },


Notifications by themselves are not very useful if something doesn’t consume them and act upon them. A naive approach would be to use email or SMS alerts and manually remove instances using the Chef user interface. Fortunately we can do better than this, and instruct machines (or scripts) to perform this (boring!) task on our behalf.

The method of consuming an SNS notification programmatically is to use Amazon Simple Queue Service (SQS) as an endpoint to SNS. The SNS topic can be configured in order to act as a producer for an SQS queue.

Once SNS and SQS are linked together, it’s just a matter of writing a consumer which is able to read EC2_INSTANCE_TERMINATE messages, and remove the node from Chef.

Given that Chef uses Ruby as language to define “recipes”, I choose to use FOG in order to leverage the SQS RESTful API.

Before retrieving the messages in the queue, authentication needs to take place.

sqs = Fog::AWS::SQS.new(
        :aws_access_key_id => access_key_id,
        :aws_secret_access_key => secret_access_key,
        :region => "eu-west-1"
       )


If the authentication is successful, the method receive_message will return a list containing  the number of messages specified as a parameter.

response = sqs.receive_message(QUEUE_URL, { 'Attributes' => [], 'MaxNumberOfMessages' => 10 })

From this point on, is a matter of parsing the JSON message, and if the event is a termination notification deleting the instance.

messages = response.body['Message']
unless messages.empty?
   messages.each do |m|
   body = JSON.parse(m['Body'])
   message = JSON.parse(body["Message"])
   if message["Event"].include? AUTOSCALING_NOTIFICATION["Terminate"]


In order to keep the code simple, the node is removed by invoking knife from the command line, so that there is no need to handle the authentication:

instance_id = message["EC2InstanceId"]
delete_node  = "knife node delete #{instance_id} -y"
delete_client= "knife client delete #{instance_id} -y"
output = `#{delete_node}`
result=$?.success?
if result == true
   result = `#{delete_client}`
end

        
Once a message is processed successfully, it can be removed from the queue:

sqs.delete_message(QUEUE_URL, m['ReceiptHandle'])

In order to work in an enterprise production environment, the script needs few additions, but this should be enough to get you started!

Alternatively, come and talk to the experts at Cloudreach.



Nicola Salvo
System Engineer
Cloudreach Limited

Thursday, 1 March 2012

Cloudreach Becomes A Google Apps Premier Enterprise Reseller



Cloudreach Limited is proud to announce that it has become a Premier Enterprise Reseller of the Google Apps™ suite of communication and collaboration tools.  This new designation from Google enables customers to more easily assess a reseller’s expertise advising on and deploying Google products.  Cloudreach has moved from an Authorized to a Premier Reseller based on their expertise and success in helping customers deploy and use Google Apps. Cloudreach is a full service Google Apps partner providing technical migration, training / change management, support and application development services around Google App Engine and Google Apps scripts.

“The Google Apps Reseller program has been a critical component of our business," said Pontus Noren, Director & Co-Founder at Cloudreach. "We’re pleased to be recognized as a Premier Reseller, and we look forward to continuing our work helping Enterprise customers take advantage of Google Apps. The market is really buzzing and the interest around Google Apps and Cloudreach is massive. Our success to date with organisations such as Trinity Mirror and London Borough of Hillingdon demonstrates that Google Apps is the enterprise solution for forward thinking corporates and local authorities alike.”

Google Apps brings simple, powerful communication and collaboration tools to organizations of any size – all hosted by Google to streamline setup, minimize maintenance, and reduce IT costs.  With Gmail (including Google email security, powered by Postini), Google Calendar, and integrated IM, users can stay connected and work together with ease. And, using Google Docs and Google Sites, which include word processing, spreadsheet, presentation and website creation tools, they can share files and collaborate in real-time, keeping versions organized and available wherever and whenever users work.

About Cloudreach
Cloudreach, with offices in London and Edinburgh, is a full service Google Apps partner able to provide consultancy to help organisation build a business, migration services to move the users and their data to Google Apps including an extensive change management capability as well as 24/7 Network Operation Centre providing Google Apps support which is tightly integrated with Google’s internal support team.

Cloudreach also provides adoption services such as Google App Engine & Google Apps Scripts development as well as Technical Account Management and consultancy services for deeper integration of Google Apps within an organisation. Customers include Trinity Mirror, AG Barr and London Borough of Hillingdon.

Google, Google Apps, Gmail, Google Talk, Google Calendar, Google Docs, Google Sites and Google Video are trademarks of Google Inc.  

Issued on behalf of Cloudreach Limited
Email: pontus.noren@cloudreach.co.uk
Tel: +44 20 7183 3893




Pontus is ready and waiting to answer your questions