Sfeng on despatches

How do you access a specific version of a parameter from AWS Systems Manager Parameter Store

Wed, 08 Apr 2020 15:53:50 +0000

This was a bit tricky to find and doesn't seem to be well documented.

```js // version number below can be a number, i.e. 3 or a label let params = { Name: "/path/to/parameter:", WithDecryption: false, }; ssm.getParameter(params, function (err, data) { if (err) console.error(err, err.stack); else console.log(data); }); ```

WARN - Missing artifact descriptor for XXX

Thu, 31 Jan 2013 20:38:48 +0000

Working on an Arquillian test deployment which had some library changes recently, I ran into the following error.

``` WARN - Missing artifact descriptor for org.javassist:javassist:jar:3.16.1-GA ```

The particular library was in the pom.xml dependency hierarchy but it was resolving to an earlier version. Maven was switched to offline mode during the tests and I had never needed this version of the library before. This meant that the local version of my maven repository did not have jar and maven emits it slightly unhelpful error. It would be better if it told us that it could not find the artifact and since its in offline mode, can't go and retrieve it.

There are two quick hacks. Add the library and version into the pom.xml and do a build. This will pull the library into your local repository and maven will be able to find it offline. You could also just take maven in the tests online by removing the goOffline() method call.

As for the cause of the issue, it stems from the way maven resolves dependencies from Arquillian in comparison to building. I had updated a library version, which now depends on the newer version of javassist. However, in considering all the other things within the pom.xml, maven brings it down to an earlier version when building.

However, the dependency resolution within the maven run through Arquillian considers a slightly different set of requirements and resolves to a later version of the lib which is not available.

Automating Code Analysis with Hudson [1115]

Wed, 28 Dec 2011 15:34:52 +0000

As part of setting up continuous integration and automated builds and source analysis, the next step is to integrate in the source analysis parts.

To this end, I installed the following plugins:

Task Scanner
FindBugs
CheckStyle
PMD

After restarting Hudson, there are a few additional configuration bits to complete.

I added an additional build step and set the goal as

``` checkstyle:checkstyle findbugs:findbugs pmd:pmd ```

I then enabled the four plugins, saved, ran a build and et voila... Just like magic

Hudson / Jenkins and Continuous Integration [1114]

Wed, 28 Dec 2011 14:51:31 +0000

Fair Warning: This is more notes for me to remember and document how to do these things rather than particularly detailed instructions. Therefore, it might be missing sections and will assume a reasonable knowledge of hudson/jenkins and not to mention the benefits of continuous integration and builds.

Installing hudson / jenkins is easy enough. I deployed as part of a pre-existing tomcat6 installation so was as simple as popping the war file into the webapps folder. Tomcat automatically started it up without issues.

I chose to have hudson use /home/hudson as its home directory. Since I am running an ubuntu system, I added a line into /etc/defaults/tomcat6. There are various other ways of doing this but it was a quick fix for me.

You of course need to make sure the directory exists. I also popped in a .m2 folder from my home directory to save it from downloading all the various jar files and included a settings.xml file with appropriate configurations.

Hudson 2.2 uses maven 3 but I use maven 3 locally as well even though the projects pom files were built for maven 2. There doesn’t seem to be any issues with this setup.

First step is to create a new job from the home page. This asks for which type a job you want to create. If you use maven and a standard source control, it is as simple as choosing the first option: Build a free-style software project.

Give it a name and you are brought to the configuration screen. There are a number of options here and I started with the basic set:

I chose Subversion for the source control management section and gave it the svn path. There is a checkout strategy as well and I chose the one to revert and update which I feel to be a bit cleaner.

I chose to poll the scm every fifteen minutes

```cron */15 * * * * ```

and saved.

Running the build pulled the code out of svn and stopped there. This was because I didn’t add a step to build / install it.

Go back into configure the job and add a maven 3 build step. This automatically selected the clean install goals. Save and build now and the project was checked out and built without issues.

Success!

There are a number of other options you can play with here but this gives you a solid starting point.

Later on, I will cover the addition of various other plugins for source analysis including findbugs and pmd.

Directed Acyclic Graphs and Executing Tasks in Order (and in Parallel) Based on Dependencies

Mon, 07 Nov 2011 23:44:36 +0000

A little while ago, there was a requirement to write a tool that could take a number of tasks each with a set of dependencies and execute them in parallel while taking the dependencies into account.

The tasks themselves were meant for data migration but that is not particularly relevant. We were writing a number of tasks which all had a set of dependencies (some of the tasks did not have any dependencies or the process could of course never start).

It was assumed that there were no cyclic dependencies (which would be error in this particular case anyway)

Bearing in mind that this was a quick and dirty tool for use three times, some of the bits in here could do with tidying up.

Each task was defined to implement the following interface

```java public interface Task extends Runnable { public String getName(); public Set getDependencies(); } ```

It should all be self explanatory. Extending the Runnable interface ensure that we can pass it into threads and other relevant bits of code. The getDependencies is expected to return the name of the tasks that it depends on.

The basic task runner which I describe below does not check if the task described in any list of dependencies actually exist. If an non-existing dependency is defined, it will likely just throw a Null Pointer Exception. I wrote this a long time ago, so don’t actually remember.

The BasicTaskRunner which we used to run the tasks implemented the TaskRunner Interface

```java public interface TaskRunner { public boolean addTask(Task task); public boolean prepare(); public boolean start(); public void waitToComplete(); } ```

The addTask method simply added it to a map from String -> Task and threw an exception in the event of a duplicate task being added in.

```java @Override public synchronized boolean addTask(Task task) { LOG.info("Adding task: " + task.getName()); if (tasks.put(task.getName(), task) != null) { throw new RuntimeException("Task with same name already exists: " + task.getName()); } return true; } ```

the prepare method just calls a method to buildGraph. This uses the jGrapht library to build a Directed Acyclic Graph

```java private boolean buildGraph() { LOG.info("Building DAG of tasks"); graph = new SimpleDirectedGraph(DefaultEdge.class); LOG.info("Adding tasks"); for (Task task: tasks.values()) { graph.addVertex(task); } LOG.info("Adding Relationships"); for (Task task: tasks.values()) { if (task.getDependencies() != null) { for (String depend: task.getDependencies()) { Task dependOnTask = tasks.get(depend); LOG.info("Adding relationship between " + task.getName() + " and " + dependOnTask.getName()); graph.addEdge(dependOnTask, task); } } } return true; } ```

So we create a simple directed graph, loop through the tasks, then each of its dependencies to create an edge, which we then add to the graph. Simple stuff really.

the start method, which actually executes the task is as follows:

```java public boolean start() { int cpus = Runtime.getRuntime().availableProcessors(); executor = new ThreadExecutor(cpus, 60, new LinkedBlockingQueue()); numTasks = graph.vertexSet().size(); LOG.info("Starting... Num Tasks: " + numTasks); startTime = System.currentTimeMillis(); scheduleTasks(); return true; } ```

As a basic algorithm, we pick up the number of available processors and use that many threads. scheduleTasks is a pseudo-recursive function whose role is to add the currently executable list of tasks into the executor to execute.

```java private void scheduleTasks() { if (graph.vertexSet().size() == 0) { executor.shutdown(); } synchronized (graph ) { Iterator iter = new TopologicalOrderIterator(graph); Set executing = new HashSet(); while(iter.hasNext()) { Task task = iter.next(); //System.out.println(task.getName()); if (graph.incomingEdgesOf(task).size() == 0 && !executing.contains(task)) { executor.execute(task); executing.add(task); } } } } ```

If there are no tasks left to execute, we shut the executor down. All being well, we add every single task in the graph that has no dependencies to be executed. The threadpool ensures that any tasks that cannot currently be executed are queued.

We use a custom version of the threadpool as follows:

```java private class ThreadExecutor extends ThreadPoolExecutor { public ThreadExecutor(int corePoolSize, long keepAliveSeconds, BlockingQueue workQueue) { super(corePoolSize, corePoolSize, keepAliveSeconds, TimeUnit.SECONDS, workQueue); } @Override protected void afterExecute(Runnable runTask, Throwable e) { super.afterExecute(runTask, e); if (e == null) { completed((Task) runTask); } else { failed((Task) runTask, e); } } } ```

The main purpose of this is to use the completed and failed callbacks to ensure that on complete, dependent tasks can be executed. On fail, we ensure that dependent tasks are not executed. The code currently does not allow for tasks that are left behind and will hang indefinitely after executing all tasks it can.

```java public void completed(Task t) { LOG.info("Completed Task: " + t.getName()); synchronized (graph) { graph.removeVertex(t); } long timeTaken = (System.currentTimeMillis() - startTime); int tasksComplete = numTasks - graph.vertexSet().size(); long timePerTask = timeTaken/tasksComplete; long totalTime = timePerTask * numTasks; long timeToComplete = timePerTask * graph.vertexSet().size(); LOG.info(" ## Tasks left: " + graph.vertexSet().size() + " ## Elapsed: " + timeTaken/1000 + " ## Est. Total " + totalTime/1000 + " ## E.T.A : " + timeToComplete/1000); scheduleTasks(); } public void failed(Task t, Throwable e) { LOG.fatal("Failed Task: " + t.getName(), e); scheduleTasks(); } ```

On completion of a task, we simply remove the task from the graph. The frees up all its dependencies to be executed. We add these tasks into the list by calling scheduleTasks again. There is nothing more for us to do when a task fails except to schedule any other tasks that can be executed. In theory, this call is redundant since any tasks that could be executed before the failure are already in the queue. Any tasks that can be completed on the completion of another item will be initiated on the completion of that task.

I hope the above makes sense and has been helpful. The code for the full class including further logging statements follows. Please bear in mind that this was hacked together over a couple of hours for something that was to be executed a grand total of three times.

```java public class BasicTaskRunner implements TaskRunner { private static final Logger LOG = Logger.getLogger(BasicTaskRunner.class); private Map tasks = new HashMap(); private DirectedGraph graph; private ThreadExecutor executor; @Override public synchronized boolean addTask(Task task) { LOG.info("Adding task: " + task.getName()); if (tasks.put(task.getName(), task) != null) { throw new RuntimeException("Task with same name already exists: " + task.getName()); } return true; } @Override public boolean prepare() { LOG.info("Preparing task runner. Num Tasks: " + tasks.size()); buildGraph(); return false; } private boolean buildGraph() { LOG.info("Building DAG of tasks"); graph = new SimpleDirectedGraph(DefaultEdge.class); LOG.info("Adding tasks"); for (Task task: tasks.values()) { graph.addVertex(task); } LOG.info("Adding Relationships"); for (Task task: tasks.values()) { if (task.getDependencies() != null) { for (String depend: task.getDependencies()) { Task dependOnTask = tasks.get(depend); LOG.info("Adding relationship between " + task.getName() + " and " + dependOnTask.getName()); graph.addEdge(dependOnTask, task); } } } return true; } public void waitToComplete() { try { executor.awaitTermination(3, TimeUnit.DAYS); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } private long startTime; private int numTasks; @Override public boolean start() { int cpus = Runtime.getRuntime().availableProcessors(); executor = new ThreadExecutor(cpus, 60, new LinkedBlockingQueue()); numTasks = graph.vertexSet().size(); LOG.info("Starting... Num Tasks: " + numTasks); startTime = System.currentTimeMillis(); scheduleTasks(); return true; } private void scheduleTasks() { if (graph.vertexSet().size() == 0) { executor.shutdown(); } synchronized (graph ) { Iterator iter = new TopologicalOrderIterator(graph); Set executing = new HashSet(); while(iter.hasNext()) { Task task = iter.next(); //System.out.println(task.getName()); if (graph.incomingEdgesOf(task).size() == 0 && !executing.contains(task)) { executor.execute(task); executing.add(task); } } } } public void completed(Task t) { LOG.info("Completed Task: " + t.getName()); synchronized (graph) { graph.removeVertex(t); } long timeTaken = (System.currentTimeMillis() - startTime); int tasksComplete = numTasks - graph.vertexSet().size(); long timePerTask = timeTaken/tasksComplete; long totalTime = timePerTask * numTasks; long timeToComplete = timePerTask * graph.vertexSet().size(); LOG.info(" ## Tasks left: " + graph.vertexSet().size() + " ## Elapsed: " + timeTaken/1000 + " ## Est. Total " + totalTime/1000 + " ## E.T.A : " + timeToComplete/1000); scheduleTasks(); } public void failed(Task t, Throwable e) { LOG.fatal("Failed Task: " + t.getName(), e); scheduleTasks(); } private class ThreadExecutor extends ThreadPoolExecutor { public ThreadExecutor(int corePoolSize, long keepAliveSeconds, BlockingQueue workQueue) { super(corePoolSize, corePoolSize, keepAliveSeconds, TimeUnit.SECONDS, workQueue); } @Override protected void beforeExecute(Thread thread, Runnable runTask) { super.beforeExecute(thread, runTask); Task task = (Task) runTask; LOG.info("Starting task: " + task.getName()); } @Override protected void afterExecute(Runnable runTask, Throwable e) { super.afterExecute(runTask, e); if (e == null) { completed((Task) runTask); } else { failed((Task) runTask, e); } } } } ```

Using CXF Interceptors to do some magic around your web service calls [1105]

Sun, 06 Nov 2011 12:24:28 +0000

We use JBossWS CXF for a heavily utilised enterprise system. It links into spring to pick up and execute beans. We have a bunch of exceptions that could get thrown.

To simplify it, the code was originally written to create an anonymous class a la Runnable which is wrapped around a try catch block. The exceptions that are thrown are then converted to a soap fault and passed back.

```java private SOAPFaultException convertToSoapException(ApplicationException e) { try { if(null == soapFactory) { soapFactory = SOAPFactory.newInstance(); } SOAPFault sf = soapFactory.createFault(); sf.setFaultString( e.getMessage() ); sf.setFaultCode( Integer.toString(e.getErrorCode()) ); return new SOAPFaultException( sf ); } catch(SOAPException soapException) { throw new RuntimeException( soapException ); } } ```

Nothing inherently wrong with this. However, there are a couple of issues with this in that each soap method is set to throw an ApplicationException and there is not further documentation of which of the subclasses are actually relevant to that method.

In a runtime environment, this is not hugely relevant. However, when generating documentation from the WSDL’s, it is.

To resolve this, we changed each method to throw their relevant exception, and wrote an interceptor to pick up the exception and convert it…

The first step was to write an interceptor which is surprisingly simple and straightforward.

```java public class ExampleSoapFaultInterceptor extends AbstractSoapInterceptor { Logger log = Logger.getLogger(getClass()); public QuarkSoapFaultInterceptor() { super(Phase.MARSHAL); } @Override public void handleMessage(SoapMessage message) throws Fault { Fault f = (Fault) message.getContent(Exception.class); Throwable cause = f.getCause(); if (cause instanceof ApplicationException) { log.info("Exception Thrown", cause); QuarkException e = (QuarkException) cause; f.setFaultCode(new QName("", String.valueOf(e.getErrorCode()))); } else { log.warn("Unexpected Exception thrown ", cause); } } } ```

The class doesn’t need to extend the AbstractSoapInterceptor but you would then have to manually implement a number of mechanisms that the abstract class provides.

The constructor simply defines where this interceptor should be inserted into as part of the chain. There is a whole bunch of different places where it can be inserted based on this. Information can be found in their documentation.

To insert this interceptor into a web service, the service needs to be annotated as follows:

```java @WebService(endpointInterface = "uk.co.kraya.example.WebService") @OutFaultInterceptors(interceptors= {"uk.co.kraya.example.interceptors.ExampleSoapFaultInterceptor"}) public class ExampleWebService implements ExampleAPI { ```

Each exception thrown from the web service will now be intercepted. You can then include any further further information in the soap fault.

In this particular case, the only thing that gets done is to update the fault code with the error code set against the exception. Any additional information can be set against the soap fault at this point. You can also log as we are doing.