tag:blogger.com,1999:blog-56051735508839747752024-03-13T06:14:59.374+00:00A Software Practitioner Still PractisingPractice makes perfect? Maybe, but I need lots of it...Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.comBlogger106125tag:blogger.com,1999:blog-5605173550883974775.post-24474092339012164602018-08-19T22:02:00.000+01:002018-08-19T22:02:05.933+01:00Woman eating waspI had a surreal experience this afternoon. Alison and I drove over to <a href="http://www.londongardensonline.org.uk/gardens-online-record.php?ID=CRO035" target="_blank">Heathfield House</a> in the London Borough of Croydon, near the top of Addington Hills. Alison had read in <a href="https://insidecroydon.com/2018/08/17/cream-teas-and-plane-spotting-at-heathfield-house-aug-19/" target="_blank">Inside Croydon</a> that there would be cream teas served on the terrace with a great view of the <a href="http://www.military-airshows.co.uk/ven10.htm" target="_blank">Biggin Hill Festival of Flight</a>.<br />
<br />
Thanks to the excellent weather and the lovely surroundings, we did have a great time - with two exceptions. Firstly, it was really a bit too far from the action (I was able to follow what was going on through binoculars, but Alison can't seem to manage binoculars because of her short-sightedness). The Red Arrows of course were easy to watch, as they trail smoke, and by being a long way off we actually got to see some extended manoeuvres in their entirety. They also came right over the house at low level a couple of times, eliciting screams of excitement from the many kids present.<br />
<br />
No, the main drawback was the pesky wasp that kept bothering Alison and taking her eyes off the flying display. I kept telling her not to try to shoo it away - "ignore it and it'll ignore you". This tactic didn't work - it settled on her arm or hand repeatedly and she said it was stinging her. I doubted it, because wasps don't sting unless threatened and they feed on sugary things and dead creatures.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/wikipedia/commons/thumb/5/59/Vespula_germanica_Richard_Bartz.jpg/220px-Vespula_germanica_Richard_Bartz.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="293" data-original-width="220" height="200" src="https://upload.wikimedia.org/wikipedia/commons/thumb/5/59/Vespula_germanica_Richard_Bartz.jpg/220px-Vespula_germanica_Richard_Bartz.jpg" width="150" /></a></div>
<br />
But I offered to swap places with her to see if it would then attack me in her place. To my amazement, this worked for a while - I kept feeling this weight on my left hand while trying to look through the binoculars, and eventually it became too irritating to ignore - it felt as if someone was applying tweezers to my skin. I flapped at it a few times to chase it away and, oh calamity, it went back to Alison.<br />
<br />
At this point, there was a lull in the display and so we went inside to get our tea and a slice of cake or scone. When we came back out, the Red Arrows display was just starting. Nobody had seized our places on the stone parapet, so we went back to them - and promptly, so did the wasp. Alison did her best to ignore it but it was persistent, so much so that when she next looked down at her left ring finger there was blood! The wasp had bitten her so hard that she was bleeding. Thankfully, having sucked up some of the blood, the infernal creature seems to have been satisfied and buzzed off. It wasn't the slightest bit interested in the jam or cream on my scones or the icing on Alison's cake.<br />
<br />
Is this a new breed of wasp - the vampire wasp?! Such things have been <a href="https://www.theguardian.com/theguardian/2012/jun/13/attack-of-the-vampire-wasps" target="_blank">reported previously</a>, but nobody appears to have taken the report seriously.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-65730999487356108172017-04-17T22:15:00.001+01:002017-04-17T22:17:59.370+01:00Drones, not roads or railways will connect Africa<div style="-webkit-padding-start: 0px; background-color: white; font-family: garamond, "new york", times, serif; font-size: 16px; margin-bottom: 0.1em; margin-top: 0.1em; padding: 0px; text-size-adjust: auto;">
Just read this brilliant article in <a href="http://www.nridigital.com/brite-magazine/issue9.html" id="yui_3_16_0_ym19_1_1492452348269_73335" style="-webkit-padding-start: 0px; background-color: transparent; color: blue; cursor: text !important; margin: 0px; outline: none; padding: 0px;">http://www.nridigital.com/brite-magazine/issue9.html</a> - a guy called Jonathan Ledgard (I don't think he's the sports commentator of the same name) is setting up "droneports" like airports for small African towns that could turn into transport hubs and much else besides. It sounds crazy, but is it crazy enough to work?</div>
Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-56804113513648624812016-07-15T21:12:00.000+01:002016-07-15T21:12:03.902+01:00How can you tell...... when politicians are lying? Their lips move!<br />
<br />
It's an old joke, but during the recent bitterly fought campaigns for and against Britain remaining in the European Union, many so-called honourable members may have inadvertently done their worst to make it true.<br />
<br />
My friend Graham Hutton thinks that politicians should be held to the same standards of honesty as company directors - who can be disqualified for five years if caught knowingly misleading the public. If you agree, please sign Graham's e-petition on the <a href="https://petition.parliament.uk/petitions/157864" target="_blank">UK Parliament web site</a>!Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-44572095996701384292016-07-14T14:47:00.001+01:002016-07-14T14:47:03.778+01:00Blockchain is dead... or is it?I have just published my first official <a href="http://blog.zuehlke.com/en/death-of-the-blockchain-exaggerated/" target="_blank">Zuhlke blog post</a>. It discusses the subject of blockchain technology and what individuals and organisations ought to be doing now.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-48584669184022007392016-07-04T15:41:00.000+01:002016-07-04T15:41:05.965+01:00Mark Twain quote - ring any bells?<span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">When the doctrine of allegiance to party can utterly up-end a man's moral constitution and make a temporary fool of him besides, what excuse are you going to offer for preaching it, teaching it, extending it, perpetuating it? Shall you say, the best good of the country demands allegiance to party? Shall you also say it demands that a man kick his truth and his conscience into the gutter, and become a mouthing lunatic, besides?</span><br />
<span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"><div>
<span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"><br /></span></div>
[from: "Consistency", paper read at the Hartford Monday Evening Club on 5 December 1887.</span><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> <i>See <a href="https://en.wikiquote.org/wiki/Mark_Twain#Quotes" target="_blank">WikiQuote</a></i>]</span>Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-77854648239904930262016-05-04T22:22:00.000+01:002016-05-04T22:22:55.816+01:00Building Successful Communities of PracticeJust got back from a stimulating evening meeting of the <a href="http://bcs-spa.org/" target="_blank">BCS SPA</a> specialist group. The presenter was <a href="http://ewebber.co.uk/" target="_blank">Emily Webber</a> on Building Successful Communities of Practice. If I understood correctly, a community of practice is a cross-functional, self-selected group of practitioners doing roughly the same tasks in different teams, business units or even organisations. <a href="http://nomad8.com/squadschaptersguilds/" target="_blank">Spotify</a> has identified almost the identical concept in its "guilds".<br />
<br />
The big takeaway for me (apart from the fact that Emily in real life doesn't look nearly as similar to <a href="http://www.sarahmillican.co.uk/" target="_blank">Sarah Millican</a> as she does in her photo) was that the mutual support provided by such a group gives people more autonomy and more mastery of their craft, both of which are strong motivating factors and promote happiness, which in turn results in demonstrably higher productivity and lower staff turnover (a <a href="https://www2.warwick.ac.uk/fac/soc/economics/staff/eproto/workingpapers/happinessproductivity.pdf" target="_blank">study</a> by Warwick University showed the productivity of happy employees to be 10-12% higher than average, while that of unhappy ones was 10% or more below). This would seem to suggest that allowing as much as one day a week to employees to exchange ideas and experiment with ways to improve their technique could be a worthwhile investment.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-1463782752147653392016-01-26T12:43:00.000+00:002016-01-27T12:23:29.232+00:00Use Vagrant to set up a Centos 7 VM in AWS EC2<h2>
Introduction</h2>
Everyone tells us that <a href="http://searchcloudcomputing.techtarget.com/definition/Infrastructure-as-Code-IAC" target="_blank">Infrastructure as Code</a> is the way to go, right? So when I was recently
asked to set up a continuous integration service for a development project, the obvious option was an approach that
allows us to script the server setup and deployment, and put the whole lot under version control. Particularly since a
colleague had already assembled a <a href="https://www.vagrantup.com/" target="_blank">Vagrant</a> file that allowed him
to deploy a Jenkins server in a VM on his workstation.<br />
<br />
However, there were a number of gotchas, which caused me a couple of days' unexpected work. I'm trying to record these
here in case anyone else benefits from my experience.<br />
<br />
One of the requirements was to run the CI server under CentOS 7, the same as the target environment, so that all tests
would run under near-identical production-environment conditions and deployment would be very simple using a tar file
containing all the dependencies. But CentOS comes with enterprise-grade security features, which sometimes get in the
way of what you want to do. Read on...<br />
<h2>
Set up Vagrant</h2>
To use Vagrant, set up a folder in your project (e.g. called "CI") and in it, create a Vagrantfile. If you have not used
Vagrant before, please work through the quick <a href="https://www.vagrantup.com/docs/getting-started/" target="_blank">Getting
Started</a> exercise to familiarise yourself with the concepts.<br />
<br />
<div>
The Vagrantfile is basically a Ruby script, so it is common practice to prefix it with:
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># -*- mode: ruby -*-</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># vi: set ft=ruby :</span></div>
</div>
<div>
<br /></div>
To use <a href="https://github.com/mitchellh/vagrant-aws" target="_blank">Vagrant with AWS</a>, first you have to
locally install the provider and the AWS box:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">vagrant plugin install vagrant-aws</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">vagrant box add dummy \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box</span><br />
<br />
The README for the Vagrant AWS provider helpfully provides a <a href="https://github.com/mitchellh/vagrant-aws#user-content-quick-start" target="_blank">starter Vagrantfile</a>
for you to copy and extend. However, we need to add a number of parameters to the AWS provider configuration and change
all the ones supplied.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.access_key_id = "**"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.secret_access_key = "****"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.session_token = "***"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.keypair_name = "*****"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.ami = "ami-e68f82fb" # CentOS 7 64-bit</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.instance_type = "t2.medium"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.region = "eu-central-1" # Pick the appropriate region</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.security_groups = [ 'sg-e430808d' ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.block_device_mapping = [{ 'DeviceName' => '/dev/sda1', 'Ebs.VolumeSize' => 50 }]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.associate_public_ip = true</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.subnet_id = "subnet-f7569b8c"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.ssh_host_attribute = :dns_name</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.tags = { 'NAME' => 'Continuous Integration' }</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws.user_data = File.read ("boothook.sh")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">override.ssh.username = "centos"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">override.ssh.pty = true</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">override.ssh.private_key_path = "~/.ec2/*****.pem"</span><br />
<br />
The private key file for your instance (or cluster) will be generated for you by AWS EC2 when you launch an instance
through the EC2 management console. The recommendation is to put this in ~/.ec2 along with any other EC2 private keys.
Do not share it via version control, or it soon won't be secret any more.<br />
<br />
The aws.tags hash allows you to set any tag values. NAME is a typical example, which allows you to identify the instance
in the EC2 management console.<br />
<br />
The aws.subnet_id should be set the same as for any EC2 instance launched manually in the same subnet.<br />
<br />
aws.block_device_mapping is only needed if you want to allocate a non-default volume size. For CI purposes, the default
volume size is probably too small if any significant amount of build history is to be kept.<br />
<br />
The aws.security-groups should list one or more security groups that you have created via the EC2 management console.
Make sure that at least the SSH port and HTTP(s) are permitted inbound. I set up an nginx server as a reverse proxy on
port 80 (see below) to allow client browsers to access multiple back-end services through the standard HTTP port.<br />
<br />
The aws.instance_type allows you to choose the size of virtual machine. You may find that t2.small is sufficient for
your needs, but if it runs out of CPU credits, it will be throttled severely (which can actually cause builds to fail
due to timeouts). While trying to perfect the Vagrantfile, however, you may wish to specify t2.micro to minimise AWS
usage charges.<br />
<br />
Notice the "boothook.sh" reference. This is based on the answer to <a href="http://stackoverflow.com/questions/17413598/vagrant-rsync-error-before-provisioning/17436026" target="_blank">a commonly encountered issue</a> with Vagrant and certain AWS AMIs. The contents of the file
are:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#!/bin/bash</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># CentOS 7 normally requires a TTY for sudo,</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># which kills Vagrant's rsync command.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># By loading this sequence of commands into</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># aws.user_data, that problem is defeated.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">SUDOERS_FILE=/etc/sudoers.d/999-vagrant-cloud-init-requiretty</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">echo "Defaults:centos !requiretty" > $SUDOERS_FILE</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">echo "Defaults:root !requiretty" >> $SUDOERS_FILE</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">chmod 440 $SUDOERS_FILE</span><br />
<br />
The Vagrantfile has a number of aws access parameters that need to be configured (shown by asterisks above). Insert the
name of your private key, which must be the one contained in the <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">override.ssh.private_key_path</span>
parameter. Navigate to the Identity and Access Management section of the AWS console. Create an IAM role for your
Vagrant execution and generate an access key. Then grant that user the AmazonEC2FullAccess permission. This allows
Vagrant to provision the virtual machine.<br />
<br />
Confusingly, this role and its keys are not inserted into the Vagrantfile at all. Instead, a session token is required.
This is how to obtain it:<br />
<br />
<ol>
<li>Download and install the <a href="http://docs.aws.amazon.com/cli/latest/userguide/installing.html" target="_blank">Amazon Command Line Interface</a>. On Mac OS X with Python and pip
already installed, this turned out to consist simply of a one-line command:<br /><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo pip install awscli</span>
</li>
<li><a href="http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html" target="_blank">Configure</a>
the command line interface:<br /><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws configure</span><br />(Leave
the default region name and default output format as "none")
</li>
<li>Request the session token with a duration of 36 hours (maximum):<br /><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">aws sts get-session-token --duration-seconds 129600</span>
</li>
</ol>
<br />
Update the Vagrantfile using the session token as well as the new key and secret key returned. After 36 hours, if you
want to deploy using this Vagrantfile again, you will have to repeat the procedure.<br />
<br />
Following the AWS provider configuration in the Vagrantfile (just above the final "end" statement in the file), specify
any further configuration steps required, e.g. synchronized folders and custom software installations (see next
section).<br />
<br />
After this, deploying the box should be straightforward (make sure you are in the same current working directory as the
Vagrantfile):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">vagrant up --provider=aws</span><br />
<br />
Note the public IP and host FQDN shown for the virtual machine in the AWS console. This is the address you will need to
access your CI application from a browser. For example, if your machine FQDN is <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ec2-54-93-105-248.eu-central-1.compute.amazonaws.com</span>, your Jenkins dashboard (assuming you went on to install Jenkins, as shown below) will be at <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">http://ec2-54-93-105-248.eu-central-1.compute.amazonaws.com/jenkins/</span>.<br />
<br />
Continue configuring your CI server manually and add these configuration
steps to the Vagrantfile if possible.<br />
<br />
To terminate the machine, use<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br class="Apple-interchange-newline" />vagrant destroy</span><br />
<h2>
Synchronize Folders</h2>
<div>
Between the <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">end</span> of the <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.provider</span>
configuration and the <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">end</span> of
the Vagrantfile, you can insert further configuration instructions. Configure SSH for folder synchronization using
the same parameters as above:
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.ssh.username = "centos"</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.ssh.pty = true</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.ssh.private_key_path = "~/.ec2/*****.pem"</span>
</div>
</div>
<div>
<br /></div>
<div>
Then specify which folders you want to synchronize to the server. Because folder synchronization precedes any shell
scripts run on the target VM, I find it best to synchronize mostly to /tmp subfolders on the target VM and then copy
or move the contents from there during the subsequent software installation. For example:
</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.synced_folder "./nginx", "/tmp/nginx", \</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">type: "rsync", create: true, owner: 'root', group: 'root'</span>
</div>
<div>
<br /></div>
<div>
where the local folder "nginx" contains a subfolder "default.d", which in turn contains "jenkins.conf" to specify
the reverse proxy configuration for Nginx to access the Jenkins server on port 8080 (see below). The copied folder
"default.d" is subsequently moved from /tmp/nginx to /etc/nginx once the Nginx software has been installed.
</div>
<h2>
Install Software</h2>
<h3>
Introduction</h3>
<div>
So-called "here documents" are a neat way to separate bits of installation script into identifiable blocks that can
be invoked from the configuration section. However, there is a "gotcha" here too - any backslashes ("\") must be
escaped ("\\"). This caught me out several times when developing sed or awk scripts in a shell and pasting them into
the Vagrantfile. (In the pieces of Vagrantfile shown in this blog post, please interpret a single backslash at the end of a line to mean a soft line wrap. Join with the following line and delete the backslash after copying! And don't insert any spaces, particularly in the middle of sed or awk scripts!)<br />
<br />
Place your here documents one after the other directly above the <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Vagrant.configure(2)</span> block.</div>
<h3>
Set up tools</h3>
<div>
The first thing is to install some tools that will be used by the subsequent installations. The time and date should
of course be set to whatever is appropriate for you.
</div>
<h4>
Here Document</h4>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">BaseConfig=<<EOF</span></span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y update</span></span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y groupinstall 'Development Tools'</span></span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install epel-release</span></span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install nano byobu bzip2 wget</span></span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install fontforge # Not required for production</span></span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo timedatectl set-timezone Europe/London</span></span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EOF</span></span>
</div>
<div>
<br /></div>
<h4>
Invocation</h4>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.provision "shell", inline: BaseBox</span>
</div>
<h3>
Set up Nginx</h3>
<div>
Note the copy command, which makes use of the folder synchronisation shown earlier.
</div>
<div>
<br /></div>
<div>
The setsebool command is required to allow Nginx to proxy HTTP or HTTPs to local TCP sockets.
</div>
<h4>
Here Document</h4>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Nginx=<<EOF</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install epel-release</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install nginx</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo cp -r /tmp/nginx/default.d/*.conf /etc/nginx/default.d/</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo setsebool httpd_can_network_connect 1 -P</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl enable nginx</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl restart nginx</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EOF</span></div>
</div>
<h4>
Invocation</h4>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.provision "shell", inline: Nginx</span>
</div>
<h3>
Set up Jenkins</h3>
<div>
This is slightly complicated by the fact that we need Jenkins to have a URL prefix - otherwise reverse-proxying
becomes next to impossible (see the sed script below). The jobs folder is relocated to /home/jenkins and
symbolically linked, which should make it easier to upgrade Jenkins later without losing build configurations and
histories. The initial set of build configurations is stored in version control and synchronised to
/tmp/jenkins/jobs by Vagrant before this installation occurs.
</div>
<div>
<br /></div>
<div>
The installation of plugins is separated into a second block. You may of course require a different selection. The
best way I have found to determine the name of the plugins to install is to install them manually once, while using
the "list-plugins" command before and afterwards to find which new plugin names have appeared. You can do this after
the Vagrant machine has been deployed by means of the following commands:
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br />vagrant ssh</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">...</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">wget http://localhost:8080/jenkins/jnlpJars/jenkins-cli.jar</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> list-plugins</span><br />
<br />
<span style="font-family: inherit;">NB if you have already enabled security on the Jenkins instance, you must log in to the Jenkins CLI before you can list the plugins.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ \<br />login </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">--username **** --password ****</span>
</div>
<h4>
Here Document</h4>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Install Java 8 update 65 because there's a bug</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># in the Jenkins update module that makes</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># signature checks fail in Java 8 > u65</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Jenkins=<<EOF</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install java-1.8.0-openjdk-1.8.0.65</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo curl -sLo /etc/yum.repos.d/jenkins.repo \</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> http://pkg.jenkins-ci.org/redhat/jenkins.repo</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo rpm --import \</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> https://jenkins-ci.org/redhat/jenkins-ci.org.key</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install jenkins</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo sed -i 's/^\\(PARAMS=.\\)/\\1--prefix=\\/jenkins /' \</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> /etc/init.d/jenkins</span></span>
</div>
<div>
<span style="font-family: courier new, courier, monospace; font-size: x-small;"># Enable Jenkins to upgrade itself automatically</span><br />
<span style="font-family: courier new, courier, monospace; font-size: x-small;">sudo chgrp -R jenkins /usr/lib/jenkins</span><br />
<span style="font-family: courier new, courier, monospace; font-size: x-small;">sudo chmod -R g+w /usr/lib/jenkins</span><br />
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl daemon-reload</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl enable jenkins.service</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl restart jenkins.service</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo mv /tmp/jenkins /home/jenkins</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo chown -R jenkins:jenkins /home/jenkins</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo ln -s /home/jenkins/jobs /var/lib/jenkins/jobs</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo chown jenkins:jenkins /var/lib/jenkins/jobs</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EOF</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">JenkinsPlugins=<<SCRIPT</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">wget http://localhost:8080/jenkins/jnlpJars/jenkins-cli.jar</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ \</span></span>
</div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> install-plugin authentication-tokens</span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin copyartifact</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin ghprb</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin git</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin git-changelog</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin git-client</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin git-parameter</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin git-tag-message</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin github</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin github-api</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin github-pullrequest</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install-plugin nodejs</span></span></span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java -jar jenkins-cli.jar -s http://localhost:8080/jenkins/ </span></span><span style="font-size: small; font-weight: normal;">\</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small; font-weight: normal;"> </span><span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">restart</span></span></span></div>
<div>
<span style="font-weight: normal;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">SCRIPT</span></span>
</div>
<h4>
Invocation</h4>
<div>
Insert some other software installations between these two in order to allow Jenkins to initialise itself before
calling it via the CLI.<br />
<br /></div>
<div>
<div style="font-weight: normal;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.provision "shell", inline: Jenkins</span>
</div>
<div style="font-weight: normal;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">...</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace; font-weight: normal;">config.vm.provision "shell", inline: JenkinsPlugins</span></span>
</div>
</div>
<h3>
Set up NodeJS</h3>
<div>
The installation of NodeJS (or node.js) under CentOS 7 is fairly straightforward, but I needed to specify the exact
versions of node, grunt and bower in order to comply with the technical policy of the project. If you don't need to
do that, just omit the version details (e.g. <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum install -y nodejs</span>).
</div>
<h4>
Here Document</h4>
<div style="font-weight: normal;">
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Node423=<<EOF</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">curl -sL https://rpm.nodesource.com/setup_4.x | sudo -E bash -</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum install -y nodejs-4.2.3-1nodesource.el7.centos.x86_64</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo npm install -g grunt-cli@0.1.13 bower@1.7.0</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EOF</span></div>
</div>
<h4>
Invocation</h4>
<div style="font-weight: normal;">
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.provision "shell", inline: Node423</span>
</div>
<div style="font-size: medium;">
<br /></div>
</div>
<h3>
Set up PostgreSQL</h3>
<div>
Here again an exact version was needed, otherwise installation could have been much more straightforward. Note the
use of double-backslashes in the awk and sed scripts.
</div>
<h4>
Here Document</h4>
<div style="font-weight: normal;">
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">PostgreSQL=<<SCRIPT</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo cp /etc/yum.repos.d/CentOS-Base.repo /tmp</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo awk '{print}; $1 ~/\\[base\\]/ || $1 ~/\\[updates\\]/'\</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">' {print "exclude=postgresql*"}' /tmp/CentOS-Base.repo \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">> /etc/yum.repos.d/CentOS-Base.repo</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y localinstall \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">'http://yum.postgresql.org/9.5/redhat/rhel-7-x86_64/'\</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">'pgdg-centos95-9.5-2.noarch.rpm'</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install postgresql95-server postgresql95-contrib</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo /usr/pgsql-9.5/bin/postgresql95-setup initdb</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo sed -i \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"s/#listen_addresses = 'localhost'/listen_addresses = '*'/" \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">/var/lib/pgsql/9.5/data/postgresql.conf</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo sed -i \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">'s/^host [^:]*$/host all all'\</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">' all md5/' \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">/var/lib/pgsql/9.5/data/pg_hba.conf</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl restart postgresql-9.5.service</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl enable postgresql-9.5.service</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo su -c "createdb mydatabase" - postgres</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo su -c "createdb mydatabase_test" - postgres</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">cat <<EOF | sudo su -c "psql mydatabase" - postgres</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CREATE USER myapp WITH PASSWORD 'myapp';</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">CREATE USER myapp_test WITH PASSWORD 'myapp';</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">GRANT USAGE ON SCHEMA public TO myapp;</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">GRANT USAGE ON SCHEMA public TO myapp_test;</span>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EOF</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">SCRIPT</span></div>
</div>
<h4>
Invocation</h4>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.provision "shell", inline: PostgreSQL</span></div>
</div>
<h3>
Set up Redis
</h3>
<div>
<h4>
Here Document</h4>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Redis=<<SCRIPT</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo yum -y install redis</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl enable redis.service</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo systemctl restart redis.service</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">SCRIPT</span></div>
<h4>
Invocation</h4>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">config.vm.provision "shell", inline: Redis</span><br />
<h2>
<span style="font-family: inherit;">Conclusion</span></h2>
</div>
</div>
<div>
I hope this has helped you in your quest for DevOps Nirvana. Please drop me a line to tell me about your experiences. I can't promise to help, but I'll lend a sympathetic ear!<br />
<br /></div>
Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-50094352577115440122015-11-17T11:41:00.001+00:002015-11-17T11:43:34.718+00:00Practical GUI testing using SikuliX<span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">Rejoice with me - after much banging of head against various walls, I have finally succeeded in building a GUI test out of <a href="http://java.com/" target="_blank">Java</a>, <a href="http://junit.org/" target="_blank">jUnit</a>, <a href="http://gradle.org/" target="_blank">Gradle</a> and <a href="http://www.sikulix.com/" target="_blank">SikuliX</a> 1.1.0</span><span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">. It starts a new game in the Mac OS Chess application, enters the opening move using drag/drop, and quits the application (answering correctly in the "save" dialog).</span><br />
<br style="background-color: white; color: #343a41; font-family: Segoe, Helvetica, Arial, sans-serif; font-size: 15px; line-height: 20px;" />
<span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">The secret was to give up on the idea of using the <a href="https://www.python.org/" target="_blank">Python</a> test script fragments created in the SikuliX IDE directly. Rather, I now translate these scripts line by line into Java; e.g. from</span><br />
<span style="background-color: white; color: #343a41; font-size: 15px; line-height: 20px;"><span style="font-family: "courier new" , "courier" , monospace;">wait("xyz.png")</span></span><br />
<span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">to</span><br />
<span style="background-color: white; color: #343a41; font-size: 15px; line-height: 20px;"><span style="font-family: "courier new" , "courier" , monospace;">screen.wait("xyz.png");</span></span><br />
<br style="background-color: white; color: #343a41; font-family: Segoe, Helvetica, Arial, sans-serif; font-size: 15px; line-height: 20px;" />
<span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">Each short Python script is converted to a simple method on an object representing the application or individual screen under test. These methods can then be invoked as fixtures from any test framework, e.g. jUnit or <a href="http://fit.c2.com/" target="_blank">Fit</a>. The resulting test suite is built with Gradle and can be run as a standalone application to test the target application end-to-end.</span><br />
<br style="background-color: white; color: #343a41; font-family: Segoe, Helvetica, Arial, sans-serif; font-size: 15px; line-height: 20px;" />
<span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">One of the tricky bits was to set the image search path correctly so that the images embedded in the jar file can be located by the SikuliX API, whether you're running in the IDE or standalone. By storing a dummy SikuliX script within the </span><span style="background-color: white; color: #343a41; font-size: 15px; line-height: 20px;"><span style="font-family: "courier new" , "courier" , monospace;">src/main/resources</span></span><span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;"> folder (e.g. </span><span style="background-color: white; color: #343a41; font-size: 15px; line-height: 20px;"><span style="font-family: "courier new" , "courier" , monospace;">images.sikuli</span></span><span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">) you can use the SikuliX IDE directly to capture and fine-tune images for use in your tests.</span><br />
<br style="background-color: white; color: #343a41; font-family: Segoe, Helvetica, Arial, sans-serif; font-size: 15px; line-height: 20px;" />
<span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">Depending on the CI environment of the project, another tricky part may be to provide both the application under test and the test suite with a graphical pseudo display on which to run, but there is quite a lot of advice in the <a href="https://github.com/RaiMan/SikuliX-2014/issues/69" target="_blank">SikuliX documentation</a> as well as on <a href="http://stackoverflow.com/questions/26032706/how-to-get-sikuli-working-in-headless-mode/26032780#26032780" target="_blank">Stack Overflow</a> about that.</span><br />
<br style="background-color: white; color: #343a41; font-family: Segoe, Helvetica, Arial, sans-serif; font-size: 15px; line-height: 20px;" />
<span style="background-color: white; color: #343a41; font-family: "segoe" , "helvetica" , "arial" , sans-serif; font-size: 15px; line-height: 20px;">Next steps:</span><br />
<br />
<ul>
<li><span style="background-color: white; color: #343a41; font-family: segoe, helvetica, arial, sans-serif; font-size: 15px; line-height: 20px;">Test this approach under Windows and Linux too</span></li>
<li><span style="background-color: white; color: #343a41; font-family: segoe, helvetica, arial, sans-serif; font-size: 15px; line-height: 20px;">Enable the use of FIT in place of jUnit (more appropriate for whole-system tests)</span></li>
<li><span style="background-color: white; color: #343a41; font-family: segoe, helvetica, arial, sans-serif; font-size: 15px; line-height: 20px;">Make use of the optional Tesseract library to read back values from the screen</span></li>
<li><span style="background-color: white; color: #343a41; font-family: segoe, helvetica, arial, sans-serif; font-size: 15px; line-height: 20px;">Build and run test suite in various CI environments</span></li>
</ul>
Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com2tag:blogger.com,1999:blog-5605173550883974775.post-67217103264423706282015-02-14T17:12:00.000+00:002015-02-14T17:44:39.068+00:00The end of FM Radio?A matter which concerns me greatly at the moment is the impending switch from FM to DAB radio, which could happen in the UK as soon as 2016. In 2013, communications minister Ed Vaizey announced a postponement of the digital switchover to "after 2015", following public pressure.<br />
<br />
Manufacturers of broadcast and receiving equipment are naturally keen to sell lots of shiny new kit. Ironically, there are technically superior alternatives to DAB already available. The technology is over 20 years old and was superseded by the incompatible DAB+ system in 2007. Canada has already abandoned DAB entirely. Here are some of the things wrong with it:<br />
<ul>
<li>The level of coverage leaves a lot to be desired - currently it is thought that 9% of the country will not have adequate DAB reception by the end of 2016</li>
<li>The quality of DAB leaves a lot to be desired. We have a couple of DAB radios in the house and usually leave them tuned to FM because the quality is better</li>
<li>DAB is not suitable for reception in a moving car</li>
<li>There is a random digitisation delay, meaning that the "pips" are not accurate enough to set a watch by. If you have radios in different rooms tuned to the same station, there's a ghastly "echo" effect because each radio has a different delay factor</li>
</ul>
Most households in the country will have to have to junk hundreds of pounds' worth of perfectly functional equipment. That's in effect a stealth tax and very bad for the environment. Many radio stations, some of which are run on a shoestring, may be forced to close down as they cannot afford the investment in new equipment.<br />
<br />
It was bad enough when analogue TV ended, but at least the digital alternative was of better quality and there were add-on Freeview receivers available at a reasonably low price to adapt existing equipment.<br />
<br />
If there's a technology that is really superior to FM radio, it should succeed on its own merits in the marketplace and not by regulation. Let's put more pressure on the government to postpone the FM switch-off indefinitely. I cannot find any on-line campaigns or petitions currently running to keep FM radio, but it could become an issue in the May 2015 UK General Election if enough of us raise the subject.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com1tag:blogger.com,1999:blog-5605173550883974775.post-78166824613018307542014-10-15T23:23:00.000+01:002014-10-15T23:23:05.326+01:00Simple, secure way to share a git repository<div class="MsoNormal">
The problem with Windows or NFS file shares is that the contents are not version-controlled and (unless you use the Windows "available off line" feature) not available while you're disconnected from the office network. However, it is very easy to set up a git repository for a team to share code and documents, if they all have at least intermittent access to the same file share.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
First install yourself a git command-line
client if you have not done so yet (see <a href="http://git-scm.com/downloads">http://git-scm.com/downloads</a>).</div>
<h4>
Creating the repository</h4>
<div class="MsoNormal">
Assuming that</div>
<div class="MsoNormal">
</div>
<ul>
<li>You already have a local git repository called "test-repo", which you want to share</li>
<li>Your local git repositories are under %HOMEDRIVE%\%HOMEPATH%\git</li>
<li>Your shared git repositories are going to be under \\myserver\myshare\git</li>
</ul>
<br />
<div class="MsoNormal">
To add a remote copy of your own git repository to the share:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>%HOMEDRIVE%</span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>cd %HOMEPATH%\git\test-repo<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>pushd "\\myserver\myshare\git"<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>git init --bare test-repo.git<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>popd<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>git remote add origin "//myserver/myshare/git/test-repo.git"<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>git push origin master<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Note that the git commands require slashes to be forward - unlike normal Windows commands.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Verify that the file <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">.git\config</span> within your local git repository looks something like this:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[core]<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> repositoryformatversion = 0<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> filemode = false<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> bare = false<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> logallrefupdates = true<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> symlinks = false<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ignorecase = true<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> hideDotFiles = dotGitOnly<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[remote "origin"]<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> url = <o:p></o:p></span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">//myserver/myshare/git/test-repo.git</span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> fetch = +refs/heads/*:refs/remotes/origin/*<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[branch "master"]<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> remote = origin<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> merge = refs/heads/master<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> rebase = true</span><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<br />
Ensure that the sections [remote "origin"] and [branch "master"] in particular are set up correctly.<br />
<h4>
Cloning the repository</h4>
<div class="MsoNormal">
Another team member can now share your code - you both push your local commits to the shared repository and pull other team members' commits from it to your local repository. Here's what each team member wishing to access the shared repository should do, assuming that they also have their local git repositories under %HOMEDRIVE%\%HOMEPATH%\git.</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>%HOMEDRIVE%</span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>cd %HOMEPATH%\git<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>git clone "//myserver/myshare/git/test-repo.git" test-repo</span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>cd test-repo<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>git pull<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">>git status<o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The status should look like this:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"># On branch master</span></div>
<div class="MsoNormal">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">nothing to commit (working directory clean)</span></div>
Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-50419755599982186752014-10-15T22:48:00.001+01:002014-10-15T22:48:10.466+01:00Making cycling safer!<a href="https://blogger.googleusercontent.com/img/proxy/AVvXsEgcMNLYyc9bkUuwtH8mn5GZ3NcPgBjbjFrCoSQqGj6qiEywz-tmCoOJgKR-vOnsodBKjN5pHMh4mpb-0kwEE9I4f3ZX6ycipqE5f3OhitPG452uQ2gabtFyAluuv2YzobOfHisIImhDuD-LeETihREv4UY4G0i4kFAFA9Y84DA-uzbcosIKUMQuZ0NRVeQy0Y064fhC_3b8EVb5ISxg-mANfEUJkf46TA=s0-d-e1-ft" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="227" src="https://blogger.googleusercontent.com/img/proxy/AVvXsEgcMNLYyc9bkUuwtH8mn5GZ3NcPgBjbjFrCoSQqGj6qiEywz-tmCoOJgKR-vOnsodBKjN5pHMh4mpb-0kwEE9I4f3ZX6ycipqE5f3OhitPG452uQ2gabtFyAluuv2YzobOfHisIImhDuD-LeETihREv4UY4G0i4kFAFA9Y84DA-uzbcosIKUMQuZ0NRVeQy0Y064fhC_3b8EVb5ISxg-mANfEUJkf46TA=s0-d-e1-ft" width="400" /></a>I have just been alerted to the existence of <a href="http://collideosco.pe/" target="_blank">Collideoscope</a> - a service that aims to collect data about cycling accidents and near misses directly from cyclists, and use it to press for improvements to road safety at accident black spots. What a brilliant idea. You can also use it to educate yourself about areas in which to take extra care when cycling.<br />
<br />Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-35719330003370634242014-01-17T15:15:00.000+00:002014-01-17T15:15:03.171+00:00Carbon - the EU's chance to shineI am really concerned that we may be causing irreversible and harmful climate change.<br />
<br />
While there is no conclusive proof that humankind is behind global warming, the evidence against us is stacking up more and more heavily.<br />
<br />
The planet has some amazing mechanisms for keeping the environment stable, such as soaking CO2 up in the oceans, but these are not infinitely flexible and we may be approaching a tipping point. As an illustration of this, consider a glass filled with water and ice cubes, standing in a warm room. The ice/water mixture stays at 0 degrees as the ice gradually turns to water, but once all the ice has melted the temperature rapidly rises.<br />
<br />
While we have not proved that our wasteful use of fossil energy is causing global warming, there is ample evidence that using a lot less won't cause any harm and may in fact stimulate the economy. So it would be logical to reduce our emissions of both greenhouse gases and waste heat as quickly as possible.<br />
<br />
More research is needed to identify renewable energy sources that are much more efficient and less environmentally damaging than wind turbines or biofuels. Better insulation and more efficient heating, manufacturing, agriculture and transport are also needed.<br />
<br />
Right now, the President of the EU Commission, Jose Manuel Barroso, is weighing a decision that could make or break our planet. His team have just days to pin down carbon emissions and renewable energy targets for Europe until 2030. It’s a very important deal that has been kept very quiet. Some EU countries, prompted by vested energy interests, are lobbying to set extremely unambitious targets. Barroso may opt to avoid a fight by playing it safe. He works in a bubble of politicians, officials and lobbyists, but he is very sensitive to what people think of him, especially now as he embarks on his last few months in the job and hopes to leave a shining legacy.<br />
<br />
Avaaz is publishing an ad in the key Brussels paper today calling on Barroso to follow the science. If supported by tens of thousands of e-mail messages from across Europe, this could just jolt him out of the bubble. You can <a href="http://www.avaaz.org/en/save_eu_climate_legacy_f/" target="_blank">read the ad and send Barroso a message today</a> that the the future of the world is in his hands.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com1tag:blogger.com,1999:blog-5605173550883974775.post-2665409069250863702013-04-08T16:45:00.001+01:002013-04-08T16:47:26.962+01:00Music by Programmers - Launch PartyJason Gorman and five of his programming chums have put together an album of original electronic music to raise money for educational programmes at <a href="http://www.bletchleypark.org/" target="_blank">Bletchley Park</a> and <a href="http://www.tnmoc.org/" target="_blank">The National Museum Of Computing</a>. The charities get every penny of the profits, split 50/50 between them.<br />
<br />
The album goes on sale in late April, and Jason's target is to sell 2,000 downloads and raise £10,000. The music industry isn't what it was, and selling music is much harder than it used to be - even for a good cause.
Between now and the release date, they're going to need as much help as they can get to spread the word.<br />
<ul>
<li>They have a web site - <a href="http://www.musicbyprogrammers.com/">http://www.musicbyprogrammers.com/</a></li>
<li>They have a Twitter handle: <a href="http://twitter.com/ProgrammerMusic">http://twitter.com/ProgrammerMusic</a></li>
<li>They have a Facebook page: <a href="https://www.facebook.com/MusicByProgrammers">https://www.facebook.com/MusicByProgrammers</a></li>
<li>They have a YouTube channel with a sneak preview: <a href="http://www.youtube.com/user/MusicByProgrammers">http://www.youtube.com/user/MusicByProgrammers</a></li>
</ul>
Please feel free to publicise their efforts by any means at your disposal.<br />
<br />
A special <a href="http://musicbyprogrammers.eventbrite.co.uk/" target="_blank">listening party to launch the album</a> on 23rd April is not to be missed. It's going to be small and intimate at a "jolly spiffy pub" near Holborn. Every penny of the £20 ticket price goes directly to Bletchley Park for maths workshops and a programming club at The National Museum Of Computing.
And every ticket holder will get a very limited edition CD with bonus tracks!!! Only 100 of these special edition CDs will be produced, so it promises to be a veritable collector's item.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com1tag:blogger.com,1999:blog-5605173550883974775.post-41935774422697615652013-04-06T19:08:00.001+01:002013-04-06T19:08:12.570+01:00GCalendar in Joomla: Daylight Savings problemWe recently launched a new <a href="http://german-church.org/london-ost/" target="_blank">Web presence for my German Church</a> using Joomla 2.5. At my suggestion, <a href="http://g4j.digital-peak.com/extensions/gcalendar.html" target="_blank">GCalendar</a> was installed to provide a searchable event calendar and an "upcoming events" module. It all worked splendidly until British Summertime began on 31 March 2013 - and suddenly all calendar events were shown starting an hour early.<br />
<br />
We spent days trying to find a way to configure Daylight Savings into the calendar. All it allowed us to do in the global settings was to choose a time zone - of course we had set it to Europe/London - but without an option of respecting Daylight Savings automatically. In despair, I eventually uninstalled the component and reinstalled it. At this point there was a new version (3.0.0) of GCalendar available, so I chose that. In the component control panel there turned out to be an options button at the top right of the screen, which we may simply have missed seeing before. In the options you can set the time zone and once I had done so, suddenly all the times appeared correctly again within the web site. Huge sighs of relief all round.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com1tag:blogger.com,1999:blog-5605173550883974775.post-53560952892064475512013-04-06T18:58:00.001+01:002013-04-06T18:58:36.861+01:00Could the "Hydrogen Age" be about to dawn?A team of researchers at Virginia Tech has used bio-engineered enzymes to <a href="http://scienceblog.com/62111/game-changer-in-alternatve-energy/" target="_blank">release high-purity hydrogen</a> from xylose in biomass and waste heat from industrial processes. Sounds too good to be true, but it was not published on 1st April.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-77550708308750503962013-02-06T14:33:00.001+00:002013-02-06T14:33:13.449+00:00Free Security Engineering textbookRoss Anderson of Cambridge has put the second edition of <i>Security Engineering – A Guide to Building Dependable Distributed Systems</i> <a href="http://www.lightbluetouchpaper.org/2013/02/04/security-engineering-now-available-free-online" target="_blank">online as a free download</a>. His explanation: it's the decent thing to do <i>and </i>it actually tends to increase sales of the printed book (borne out by sales of the first edition). Music recording industry, please take note!Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-27945795018857789562013-01-16T08:57:00.001+00:002013-01-16T08:57:07.155+00:00libharu-2.2.1.tar.gz disappearsOvernight, my project's build suddenly failed. The makefile is supposed to download and rebuild the libharu pdf generation library whenever it creates a clean build. It turned out that the <a href="http://libharu.org/" target="_blank">libharu project</a> has moved all its files on to Github and this much-used stable release has not been tagged.<br />
<div>
<br /></div>
<div>
Luckily I found a mirror under the <a href="http://fedoraproject.org/" target="_blank">Fedora</a> project's <a href="http://pkgs.fedoraproject.org/repo/pkgs/libharu/libharu-2.2.1.tar.gz/63da8e087744f1d6cf16f9501b1cb766/libharu-2.2.1.tar.gz" target="_blank">repository</a> - thanks Fedora!</div>
Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-67427883022760552662012-10-17T21:48:00.001+01:002012-10-17T21:48:44.470+01:00<h2>
Doomed to oblivion: a very promising cure for cancer</h2>
Read about <a href="http://icancer.org.uk/the-science-behind-icancer/" target="_blank">research by Professors Justyna Leja and Magnus Essand</a> into a virus that can destroy pancreatic cancer cells without noticeable side effects. Then share their frustration that no company or government is prepared contribute the ridiculously small sum of £2M needed to fund clinical trials. Without these, the treatment will never be licensed for use in humans.<br />
<br />
At the time of writing, the <a href="http://icancer.org.uk/help-crowdfund-the-anti-cancer-virus-that-could-save-thousands/" target="_blank">campaign to crowdsource</a> the necessary funding has been running for almost two weeks, with a further four to go - but less than 10% of the total has been raised so far. This needs a massive push now! Please alert your friends.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-75641794978806058382012-04-30T11:17:00.001+01:002012-04-30T11:18:52.963+01:00Moving Mail to a New LaptopMy wife was getting fed up with her eMac running slower and slower, plus incompatibility between her MS Office X and the MS Office 2010 in use at her company, so for her birthday she got a Windows 7 laptop. Until the weekend just gone, however, she didn't make much use of it because it was beyond her ability to move her mail account from Entourage to Outlook and to move all her files across. The address book was a particular problem.<br />
<br />
Eventually we found ways to achieve almost all these migrations. The document files were the easiest part - local area network connection, turn on Windows sharing on the Mac, just drag the files over (we had to retry a few times because the connection reset while copying long files).<br />
<br />
To migrate the mail messages, we decided to use Thunderbird instead of Outlook (this PC doesn't connect to a corporate Exchange server). Drag each folder from Entourage to a Finder folder or the desktop. This exports the messages in the form of a .mbox file. Rename to remove the extension and copy them to Thunderbird's Mail folder in the current profile.<br />
<br />
The mail addresses turned out to be rather a tougher proposition. You can export each address card to a .vcf file, but Thunderbird found no useful information in them. It dawned on me later that this might be a line-ending issue - we could have run mac2unix and then unix2dos before importing. But by then we had solved the problem in a different way.<br />
<br />
Entourage allows you to drag address cards to the Mac's native address book application. This was synchronized with <a href="http://www.plaxo.com/" target="_blank">Plaxo</a> using my wife's account. Plaxo supports export of the address book in LDIF format, which Thunderbird was able to import without problems. If you can't use this route, you can also export the entire Mac address book in a single operation to .vcf (or possibly even to LDIF). I suggest adjusting the line endings to Windows conventions before attempting to import to Thunderbird.<br />
<br />
Then it turned out that the vast majority of e-mail addresses that she needed were not in either address book. Entourage just maintains a cache of recently used addresses and we could find no way to export all of these into any sort of useful file format. To the rescue came <a href="http://aiminstitute.org/blog/2010/06/extracting-email-addresses-from-mbox-files" target="_blank">Adam Haeder</a> with a simple shell script to extract plausible e-mail addresses from mbox files. I was able to adapt it for use under <a href="http://www.cygwin.com/" target="_blank">Cygwin</a> and enhance it a bit to cope with non-US addresses and escaped newlines and spaces. The resulting tab-delimited file was suitable for import to Thunderbird.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1GHytawL4FhzijhFNUYYDn2aj7BuVoJ0ycZvnQLfUkF6pR1cft_vIdSFCLfLNe3bIJyIq4WdYodDjyW-r0h6lAj_PHpMrKxHR28a1BtSBgcRUsF-_A980pAUDM4D-xYxpWP9jWaSGJihB/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> #!/bin/bash
# This script will parse an mbox file, displaying all of the From: email addresses, removing ones that
# are from postmaster, mail admins, etc
FILE=$1
if [ ! -r $FILE ]; then
if [ -r /var/spool/mail/$FILE ]; then
FILE="/var/spool/mail/$FILE"
else
echo "Sorry! Neither $FILE nor /var/spool/mail/$FILE exists, or I can't read them"
exit
fi
fi
# Using cat to read the input means you can run this over many mbox files simultaneously
# dos2unix and mac2unix standardise line endings to LF only
# First sed script joins lines that have been split with a "=" at line end
# grep isolates lines starting with "From:" and egrep -vi rejects all non-human addresses
# Next grep discards any lines that contain no email address (@)
# The next sed script turns =20 into spaces and discards trailing spaces and "From:"
# The next converts [mailto:x@y.z] to <x@y.z> form
# Any bare SMTP addresses are converted to "x@y.z <x@y.z>"
# Any SMTP addresses with no friendly name but in "<>" delimiters are converted similarly
# Finally the "<>" delimiters are removed and replaced with a tab separator
# Result is sorted uniquely (ignore case & leading blanks) and converted to DOS line endings
cat $FILE | dos2unix.exe | mac2unix.exe |\
sed '/=$/{N;s/=\n//}' |\
grep "^From:" | egrep -vi \
"(postoffice|\
postman|\
administrator|\
bounce|\
MAILER-DAEMON|\
postmaster|\
Mail Administrator|\
Auto-reply|\
out of office|\
Mail Delivery System|\
Email Engine|\
Mail Delivery Subsystem|\
Mail.Administrator|\
non.deliverable)" |\
grep '@' |\
sed 's/=20/ /g;s/\s*$//;s/^From:\s*//' |\
sed 's/\[mailto:\(.*\)\]/<\1>/g;s/"//g' |\
sed '/^[^<]*$/s/^\(.*\)$/\1 <\1>/' |\
sed 's/^<\(.*\)>$/\1 <\1>/' |\
sed 's/^\(.*\S\)\s*<\(.*\)>/\1\t\2/' |\
sort -ubf | unix2dos.exe
</code></pre>
<br />Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-38180310412190206692012-03-14T11:11:00.001+00:002012-03-14T11:28:52.790+00:00BDD for JSP in Eclipse with Groovy Webtest<div xmlns="http://www.w3.org/1999/xhtml">
Pak Wing and I needed to put together a rapid prototype of a document generator that delivers web pages using custom templates together with various back-end sources of information. The first thing we did, of course, was to sketch out a test plan to identify the capabilities of our system. A simple first test was to see that the correct page title would be generated in response to a GET with varying parameters in the query string, including an invalid case.<br />
<br />
The next question was how to execute the tests. My previous experience with <a href="http://webtest.canoo.com/webtest/manual/WebTestHome.html" target="_blank">Canoo Webtest</a> had been favourable, so I was keen on this approach - but not on the traditional <a href="http://www.w3.org/XML/" target="_blank">XML</a> notation that Webtest uses. Since it has become feasible to use <a href="http://groovy.codehaus.org/" target="_blank">Groovy</a> as the test scripting language, we decided to use this as it can support the <a href="http://behaviour-driven.org/" target="_blank">behaviour-driven</a> approach (where test cases are expressed in the "given, when, then" form), is a more compact notation than XML and easier for our stakeholders to read.<br />
<br />
However, Canoo's manuals are a little light on detail about how to get all this to work in an <a href="http://www.eclipse.org/" target="_blank">Eclipse</a> environment, so I'm recording my findings here as I go along.<br />
<br />
<b><big>Software to Install</big></b><br />
Packages:<br />
<ul>
<li><a href="http://www.oracle.com/technetwork/java/javase/downloads/index.html" target="_blank">Java SE JDK</a></li>
<li><a href="http://www.eclipse.org/" target="_blank">Eclipse</a></li>
<li><a href="http://groovy.codehaus.org/Download" target="_blank">Groovy</a></li>
<li><a href="http://webtest.canoo.com/webtest/manual/Downloads.html" target="_blank">Canoo Webtest</a></li>
<li>An application server - in our case, we wanted to use JSP, so <a href="http://tomcat.apache.org/download-70.cgi" target="_blank">Tomcat</a> version 7 was the obvious choice. As we're developing under Windows 7, we chose the Windows Service installer version (but note that the service should not be started - Eclipse will run the server when you're ready to deploy the application)</li>
</ul>
Eclipse plug-ins:<br />
<ul>
<li>Eclipse Web Developer Tools</li>
<li>Eclipse XML Editors and Tools</li>
<li>Groovy-Eclipse Feature (optionally plus sources)</li>
<li>Web Page Editor (optional)</li>
<li>Don't forget integration with your favourite Software Configuration Management system</li>
</ul>
I hope that was everything - please drop me a comment if you find you're missing some vital component that I've forgotten to mention.<br />
<br />
<b><big>Starting your Project</big></b><br />
After a number of experiments, we found that the most straightforward approach was to create a new Dynamic Web Project and to convert it to a Groovy project afterwards, using the Groovy entry in the context menu. You can also convert to a Maven project subsequently, if that is your preferred build system and you have the M2E (Eclipse Maven Project) plugin installed.<br />
<br />
I like Maven's organisation of source folders into src/main and src/test. Put your Java files (e.g. servlets) in src/main/java and test scripts in src/test/groovy. HTML and JSP files go into src/main/resources. Set up the Java Build Path appropriately.<br />
<br />
It is standard practice to name the package for a JUnit test script identical to the package of the class under test. We followed this practice also for Groovy test scripts. A Groovy Webtest script is in fact a thinly disguised <a href="http://ant.apache.org/" target="_blank">Ant</a> build script, but you run it as a <a href="http://www.junit.org/" target="_blank">JUnit</a> test (while avoiding all the tedium of writing JUnit test cases in Java). The Ant libraries are supplied with Webtest.<br />
<br />
<big><b>Running Your First Test</b></big><br />
Here's my version of the simple test script shown on the Canoo home page. Note the addition of firewall configuration:<br />
<br />
<iframe src="http://pastebin.com/embed_iframe.php?i=ddBwbSeF" style="border: none; width: 100%;"></iframe><br />
<br />
If you paste this into your own Eclipse project, you will immediately be informed that the import "com.canoo.webtest.WebtestCase" cannot be resolved. To fix this, open the Java Build Path configuration, click on the Libraries tab and add a new User Library. I called it WEBTEST_LIBS. To define this User Library, click "Add JARs", navigate to the lib folder under your Webtest installation directory, highlight all jar files you find there and click OK.<br />
<br />
Obviously you should amend the firewall configuration in the above constructor function TestGroovyWebTest(<span class="kw2"></span>) as appropriate. <br />
<br />
To run the test script, simply right-click the Groovy file, select "Run as..." and click "JUnit Test". You should get both the standard JUnit output of a green bar (hooray!) and a Webtest monitor window that shows the test in progress, followed by a result screen in your web browser.<br />
<br />
<big><b>Create and Deploy a Dummy Application</b></big><br />
For the next stage, we wanted the Groovy test script to actually interact with the application we were developing.<br />
<br />
First configure your server. In Eclipse, go to the View menu, select "Show View", expand the "Server" section and select "Servers". If this is not available, you probably have not installed the Eclipse Web Developer Tools feature correctly.<br />
<br />
Right-click in the Servers panel and select "New". Add the server (you should find it in a pull-down list). Configure the correct run-time environment for the application server you have installed, and name it (probably "localhost"). Click "Finish" (don't configure any apps at this stage).<br />
<br />
Create a very simple index.jsp page under src/main/resources:<br />
<br />
<iframe src="http://pastebin.com/embed_iframe.php?i=45ZMp6st" style="border: none; width: 100%;"></iframe><br />
<br />
Now you can right-click the JSP file, select "Run as..." and click "Run on server". The output ought to appear in a new edit panel.<br />
<br />
You can now add a test case to invoke your new app and check that it has produced the right page:<br />
<br />
<iframe src="http://pastebin.com/embed_iframe.php?i=adgUeLMW" style="border: none; width: 100%;"></iframe><br />
<br />
<b><big>Gotchas</big></b><br />
There are a few problems to look out for:<br />
<ul>
<li>If you collaborate with someone else on your project, and their Eclipse and other configurations do not match yours precisely, you may find that some of the build path configuration has to be changed after they import your project. To minimise this, make sure that for example you configure "workspace default" as the JRE for the project.</li>
<li>In one case, we found that Webtests could not be run more than once. The reason was that each run produces a report under the same folder into which the Java and Groovy compilers place class files. For some reason, with certain installations of the Tomcat server, the synchronisation mechanism meant that the test reports were being deployed along with the application, which meant that Webtest could not delete them before running the next test. In these cases, we found that the only remedy was to restart Eclipse and then start Tomcat again - too tedious in the long run. It might be less hassle just to run the Groovy test script outside Eclipse, but we have not tried this.</li>
</ul>
<br />
<br />
<div class="zemanta-pixie">
<img alt="" class="zemanta-pixie-img" src="http://img.zemanta.com/pixy.gif?x-id=0f431fb8-9dd0-8dc0-ad23-507c0aa4bafc" /></div>
</div>Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com1tag:blogger.com,1999:blog-5605173550883974775.post-27300629604224838172012-02-19T20:44:00.000+00:002012-02-19T20:44:03.526+00:00Feed the worldRespect to my daughter Ruth for deciding to give up meat and dairy products for Lent. Her reasons are explained in her <a href="http://redwalldiet.blogspot.com/" target="_blank">blog</a> and she promises to post a series of mouthwatering recipes there.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-85148340010585799002012-01-31T11:04:00.000+00:002012-01-31T11:05:34.784+00:00Ubuntu Linux desktop 11.10 proxy settings solvedWatch out for system proxy settings in <a href="http://www.ubuntu.com/ubuntu/whats-new" target="_blank">Ubuntu Desktop 11.10</a> (at least in its <a href="http://www.vmware.com/appliances/directory/va/1263893" target="_blank">LXDE</a> guise, which is what I am running in a VM). The control panel "Network" has disappeared and even if you install it using apt or Synaptic you'll find that it doesn't have any proxy options. gconf does, but seems to have no effect. Answer is to edit /usr/share/applications/chromium-browser.desktop and append<br />
<br />
--proxy-pac-url="<a href="http://example.com/" target="_blank">http://example.com/</a>"<br />
<br />
to the Exec line.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com1tag:blogger.com,1999:blog-5605173550883974775.post-62227845923446739892011-09-24T10:17:00.000+01:002011-09-24T10:22:54.997+01:00Fun with Robots, Arduino and AndroidTogether with Mike Hogg from my firm, Zühlke Engineering, I recently developed a demonstration project over a three-day training exercise.
We purchased a <a href="http://robotshop.com/eu/dfrobotshop-rover-tracked-robot-bluetooth-kit.html">simple robot</a> and added a set of three infra-red sensors capable of detecting a black line marked out on the ground with electrician's tape. We had to develop a line-sensing algorithm and control logic to follow the line and stop at stations (a short length of tape at right angles to the line) until instructed to move on.<br />
<br />
We followed a <a href="http://en.wikipedia.org/wiki/Test-driven_development">test-driven development</a> (TDD) approach for C++ using <a href="http://www.cpputest.org/">CppUTest</a>, as described by <a href="http://pragprog.com/book/jgade/test-driven-development-for-embedded-c">James Grenning</a>, to create a <a href="http://objectvalue.com/articles/CppFiniteStateMachine.html">state machine</a> to turn sensor readings into motor control instructions (with hindsight, we might have saved time by resorting to the <a href="http://ehiti.de/machine_objects/">Machine Objects library</a>). To my surprise, once we loaded the finished firmware to the <a href="http://www.arduino.cc/">Arduino</a> board, it worked very well with only a bit of tweaking of the motor speed settings corresponding to "hard left", "slight left" etc.<br />
<br />
Then we linked this with an application we developed for an <a href="http://developer.android.com/">Android</a> tablet computer (Motorola Xoom) that could be used to instruct the robot to go to any selected station, pick up/drop a "payload" (only conceptually) and return to base. The two components communicated by <a href="http://www.bluetooth.com/">Bluetooth</a> radio link.<br />
<br />
Finally, the tablet app was linked via a <a href="http://en.wikipedia.org/wiki/Representational_state_transfer">RESTful Web Service</a> interface to an "enterprise" system developed by other colleagues on the training course, that controlled the whole delivery network and knew which consignments had to be delivered to what stations. The tablet would send a message to the enterprise server telling it where the robot had arrived, and receive back an instruction containing the name of the next station to move to. The robot would then move off automatically after 5 seconds.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-kspgnGt7-4k/Tn2Nz8VcLFI/AAAAAAAADU8/PebPiop8QwA/s1600/DSC01614.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="http://1.bp.blogspot.com/-kspgnGt7-4k/Tn2Nz8VcLFI/AAAAAAAADU8/PebPiop8QwA/s320/DSC01614.JPG" width="320" /></a></div>
<br />
The whole thing was enormous fun (as well as very instructive) and was demonstrated successfully to the entire group (see photo).<br />
<br />
<a href="http://www.codemanship.co.uk/">Jason Gorman</a> and <a href="http://research.microsoft.com/~simonpj">Simon Peyton Jones</a>, among others, have recently been at pains to point out the shortcomings of <a href="http://www.computingatschool.org.uk/">IT education in the UK</a>. Thinking about the above exercise, it occurred to me that it is just the sort of project that could engender enthusiasm for the subject from a wide range of school students. It involves lots of varied tasks from assembling the hardware to designing the user interface, and of course the devising of suitable protocols between all the components.<br />
<br />
Simon Peyton Jones of Microsoft Research will be presenting an <a href="http://bcs-spa.org/cgi-bin/view/SPA/ComputingAtSchool">interactive talk</a> about Computing At School on 2nd November in London, and the following month (6 December) Mike and I plan to do a session on the Robot Shop exercise at the same location. Anyone interested will be most welcome.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com2tag:blogger.com,1999:blog-5605173550883974775.post-25828206479210249272011-08-10T14:22:00.002+01:002011-08-10T14:25:00.298+01:00Generate test data for Java development easily<div xmlns='http://www.w3.org/1999/xhtml'>A colleague has just pointed out the release of <a href='http://www.andygibson.net/blog/article/generate-test-data-with-datafactory/' target='_blank'>Andy Gibson's DataFactory library</a> as a Maven artifact. It generates arbitrarily long lists of random names and addresses, dates, date ranges (with arbitrary constraints), random selections from lists, numbers, strings, Internet addresses etc. etc.<br/><br/><div class='zemanta-pixie'><img src='http://img.zemanta.com/pixy.gif?x-id=79e9dab2-2060-834f-83ea-2fc0ae0b6c4d' alt='' class='zemanta-pixie-img'/></div></div>Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com0tag:blogger.com,1999:blog-5605173550883974775.post-68650526491142667692011-06-19T15:19:00.006+01:002011-06-19T15:34:57.451+01:00Inspiring the next generation of developers<a href="http://http://codemanship.co.uk/parlezuml/blog/?postid=1039">Jason Gorman</a> has published a thought-provoking opinion piece asking why schools are doing so little to promote IT as an enticing career option and to equip kids to take advantage of it. His analysis points to a lack of suitably qualified teachers (so what's new?) and a commensurate lack of ambition on the part of exam boards to make the syllabus industry-relevant.<br /><br />Do <a href="http://www.codemanship.co.uk/parlezuml/contact.htm">get in touch</a> with Jason to take part in a summit at <a href="http://www.bletchleypark.org.uk/">Bletchley Park</a> (where else?) on 25th August 2011 to identify practical measures that can be taken to improve this dire situation.<br /><br />I think this is an issue the <a href="http://bcs.org/">BCS</a> should be addressing with all dispatch instead of tying itself in knots over some elusive ideal of making all IT practitioners professionally qualified. Some of the most proficient software practitioners I know have no professional qualifications at all.Immo Hünekehttp://www.blogger.com/profile/17107000566759440656noreply@blogger.com5