mv

Maven Copy Dependencies

2011-04-28T07:07:00.000-07:00

The last project I started, which is really a web app prototype, I decided not to use Maven and go with a standard Eclipse project layout instead. It's easy to run that way from the IDE. However, lots of libraries are Mavenized these days, so it's a tedious process to gather all your dependent jars. I used to do this by trying to run it, and fixing the situation whenever encountering a ClassNotFoundException. It's a tedious job, especially if you use a handful of libraries. That's when I stumbled across the dependency:copy-dependencies goal. This is handy, because Maven will get all the jars you need for a particular library. Anyway, a big time saver.

Facebook Style Time Ago as a JSF Converter

2011-03-04T03:55:00.000-08:00

Here is a way to render date-time values the way Facebook does (4 hours ago, 2 days ago, etc.) The algorithm was taken from a Snipplr post, which was written in PHP, so I had to convert it into Java. So this is what the JSF converter looks like:

public class RelativeDateTimeConverter implements Converter {

private static final String[] periods = {"second", "minute", "hour", "day", "week", "month", "year", "decade"};
private static final double[] lengths = {60, 60, 24, 7, 4.35, 12, 10};

@Override
public Object getAsObject(FacesContext context, UIComponent component, String param) {
// Not implemented because not needed
}

@Override
public String getAsString(FacesContext context, UIComponent component, Object value) {

Date now = new Date();
Date date = (Date) value;
long diff = (now.getTime() - date.getTime()) / 1000; // diff in seconds

int j = 0;
for (int i=0; diff >= lengths[i] && i < lengths.length -1; i++) {
j = i;
diff = (long) (diff / lengths[i]);
}

String period = periods[j + 1];
if (diff != 1) {
period = period + "s";
}

return diff + " " + period + " ago";
}
}

You have to register the converter in the faces-config.xml file, like such:

RelativeDateTimeConverter

com.ferdinand.datebook.util.RelativeDateTimeConverter

And once that's done it's easy to use, like such (took the prefixes out because for some reason they won't render in the blog :-):

Dumb Guy with a Smart Phone

2011-02-27T17:29:00.000-08:00

That's how I feel whenever I leave that damn thing at home. :-)

Multi-lingual

2011-02-25T10:52:00.000-08:00

I was trying to come up with a list of languages I touched in the past, and in chronological order, it would look something like this:

0. Dutch: I was born and raised in Holland, so Dutch (Nederlands) is my main programming language of choice.

1. German: It's actually very similar to Dutch, it's a germanic language as well.

2. English: Yes, very simple language to learn. Seems less verbose than Dutch in some ways.

3. French: Got to learn this in school for 3 years I think. I was pretty good at pronunciation, but too many exceptions than rules.

4. "Multinational English": Living in NYC, you get to work with a lot of different people from all over, each bringing our own English pronunciation. I've worked with people from Vietnam, India, Ukraine, Russia, Germany, France, Costa Rica and it all sounds a little different.

5. Basic: Back in the 80s, when there was such a thing like a home computer, that's what you had to program with. Stored on a tape recorder for retrieval :-)

6. Pascal: We got to do some programming with this in university. This was all in a DOS kind of environment, although later I think they came up with Turbo Pascal on Windows.

7. Visual Basic: This was another assignment I had to do in university, to program a GUI for a WYSIWYG text editor. Drag-and-drop building displays with programmable event handlers.

8. Lisp: Don't remember too much about this one, other than that we used it inside AutoCAD to create macros. Running on Apollo workstations and later on Sun Solaris. Must have been mid nineties.

9. LotusScript: I have spend a lot of time coding in LS while doing Notes development. This was late nineties, when "groupware" was hot. Later on, Lotus also started to support Java and JavaScript within its platform.

10. JavaScript: Once I started doing web applications in Lotus Notes & Domino, JavaScript was the next logical thing. Simple things like form validation etc, nothing too fancy.

11. HTML: One can argue that HTML is not really a language, but when you go DHTML is becomes a lot more programming. Mostly did some nifty drop-down menus etc and rotating banners.

12. Java: Java was another language that got introduced into Lotus Notes. But at a certain point in time, we went all out Java, with a portal, a CMS, search and IDM all on the same platform. There was no escaping Java with IBM as your vendor.

13. MS ASP: My company used to do a lot of stuff for Intel, and the main agency was a big Microsoft shop. So everything was on in Active Service Pages on IIS, with SQL 2000 for relational storage. It was great, because simple to use. Not much object orientation or code reuse though...

14. Python: This is a weird one. I got to program in it when working on a Vignette CMS for collaboration. It's an elegant language, easy to read. I've also used it recently to export a SQLite database into XML documents.

15. DotNet & C#: This one I liked, but I only worked with .NET for 10 months or so. I was able to clobber together a time tracking application, hand-coded AJAX calls included :-) Visual Studio is great, and C# is easy to learn when you know java.

16. Groovy: This is a nice one, which I got to play with a little when I was checking out Groovy. It's an elegant language, more compact than java.

Alfresco Actions & Multi-Threading

2011-02-20T11:55:00.000-08:00

For a particular project we ingest large amounts of XML and binary data into Alfresco on a continuous basis. We use a "drop box" approach, where spaces in Alfresco are assigned a content rule to call a certain action. The action attempts to file a node into a certain repository path, and tries to create the path structure if it doesn't exist. It works great with a single request, but when you start dropping multiple files at the same time, we run into concurrency issues. For dropping we use a custom web-script that takes the file as multi-part form-data stream in a form POST.

It seems that each action invocation is wrapped in its own implicit transaction, with its own private memory space. The problem arises with concurrent drops, when thread #1 attempts to create a node that was already created by thread #2, more or less while both transactions are running. Both transactions are not aware of each other's state.

This last statement is not completely true. It seems that you can find a NodeRef through the SearchService with a Lucene query, even if it was created in another transaction. Unfortunately this is not the case for the NodeService, which you need to rely on for a lot of the work. It chokes on NodeRef instances found by the SearchService.

It's possible to synchronize all action code, but this still doesn't solve the duplicate node issue, which is what happens when 2 threads try to create the same node in isolation from each other. You simply don't seem to have control over which thread will be committed first, even if you use your own non-propagating transactions within the synchronized block.

Even a RetryingTransactionHelper CallBack was implemented, but to no avail. One thread simply cannot see the state of the other thread's transaction, at least the NodeService can't. It would be cool if you could refresh a current transaction with the latest changes to the repo.

Anyway, this whole ordeal took more than 3 days with multiple engineers looking at the issue at the same time. We considered using a scheduled Alfresco Action as an alternative to the content rules, but decided against it. Now we're implementing the action as a remote call using RAAr, a java API for remote communication with Alfresco ECM.

The cool thing about RAAr is that it follows the Alfresco Foundation API to the letter, so in theory you can take Alfresco compliant code and drop use it in conjunction with RAAr without a problem.

All method call with RAAr are are remote, and handled by a web script. Every call is also wrapped in its own transaction, which is... nice :-) The only downside is that it will be slower, but if it's more reliable I don't care too much about that.

Ok, after this shameless plug, a few last words. Although Alfresco is highly transactional, it may not always work as expected using the Alfresco Foundation API. There's probably a bug, or a possibility for improvement, but it would be time consuming. A remote ingestion approach, while slower, is a more reliable way of managing new and updated content inside the repo.

That's it for now.

Windows vs Unix File System

2011-02-18T06:49:00.000-08:00

I'm running a project at the moment, where we need to publish files to a Windows NTFS file system.

Ran into some issues, because some of the file paths were getting longer than 256 characters. At least it seems an issue extracting such long file names from a ZIP or TAR file. Not sure if there are difference between Windows XP and Windows 2008 Server. It seems possible to go over the limitation of 256 chars on the server, when you copy files directly over the file system (SMB in this case).

Also saw some issues with the use of reserved words on the Windows FS. The following words are reserved for devices on NTFS: CON, PRN, AUX, CLOCK$, NUL, COM0, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT0, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. So as a solution we had to wrap them.

Very odd, and in that respect we're a lot better off on Linux and Mac OS.

Writing custom actions for Alfresco

2011-02-02T07:07:00.000-08:00

As we all know, writing code is an iterative process. You never get it right until you tried several iterations, maybe up to 20 depending on complexity. Too bad java code needs to be compiled, which is particularly annoying for deploying and running Alfresco extensions.

So here is my iterative development cycle:

1.) Write some code (in Eclipse, mostly)
2.) ./alfresco.sh stop
3.) mvn package
4.) java -jar bin/alfresco-mmt.jar install ~/path/to/my/extension.amp tomcat/webapps/alfresco.war -force -nobackup;rm -r tomcat/webapps/alfresco;./alfresco.sh start
5.) Test and see what happens

Step 4 as you can see is the killer, as starting Alfresco takes 90 seconds (I want an SSD, please). Enough to keep checking my e-mail for no reason. Maybe they should do something cool like OSGI next time, so we can deploy that stuff on-the-fly into the box. Anything to make my cycle go faster :-)

Alfresco Documentation

2011-02-01T08:04:00.000-08:00

When you do Alfresco development or integration, the wiki is your friend. Don't try to use the wiki's search though, I usually use Google with better results (site:wiki.alfresco.com keyword1 keyword2 etc.) Your other friend will be the Alfresco source code, since the wiki won't tell you everything. I guess this is true for any open source product. Once I have the source code, all I need is Mac OS's Spotlight and a text editor and that's it :-)

iPod Quality

2011-01-22T07:27:00.000-08:00

So the other day I accidentally dropped my iPod in a big puddle of water on the corner of the street. It got completely submerged for maybe 5 seconds. It was actually on when it happened, and after I picked it back up, I saw the screen slowly die :-( Ok, it's about six years old, but still... Anyway, I found this post which basically said to dry it out good, and it could still come back to life, and it did!!! Pretty amazing, although it doesn't beat the story of the guy who had his iPod in a commercial washer for 40 minutes.

Recreating an Alfresco instance

2010-05-19T07:27:00.001-07:00

Sometimes Alfresco becomes a little messed up, and you may not have a choice other than to recreate it. This can be done by deleting everything in the alf_data directory and recreating the (empty) database with the mysql command line client:

drop database alfresco;

create database alfresco default character set utf8 collate utf8_bin;

Once Alfresco starts up, it will recognize that the database is empty, and it will kick off a process to recreate the schema. However, be sure to restart Mysql before starting Alfresco. It seems that foreign key names get stuck in cache, even after the database holding the keys gets dropped. If you don't restart Mysql, you may see messages like this in the catalina.out:

10:00:55,707 INFO [domain.schema.SchemaBootstrap] Schema managed by database dialect org.hibernate.dialect.MySQLInnoDBDialect.

10:00:55,857 INFO [domain.schema.SchemaBootstrap] Executing database script /Users/mverkaik/servers/alfresco-3.2.1/tomcat/temp/Alfresco/AlfrescoSchema-MySQLInnoDBDialect-Update-2896636379215681530.sql (Generated).

10:01:03,872 ERROR [domain.schema.SchemaBootstrap] Statement execution failed:

SQL: alter table JBPM_BYTEBLOCK

add index FK_BYTEBLOCK_FILE (PROCESSFILE_),

add constraint FK_BYTEBLOCK_FILE

foreign key (PROCESSFILE_)

references JBPM_BYTEARRAY (ID_)

Error: Duplicate key name 'FK_BYTEBLOCK_FILE'

File: /Users/mverkaik/servers/alfresco-3.2.1/tomcat/temp/Alfresco/AlfrescoSchema-MySQLInnoDBDialect-Update-2896636379215681530.sql

Line: 813

10:01:03,873 ERROR [domain.schema.SchemaBootstrap] Schema auto-update failed

Alfresco Workflow

2010-05-14T07:07:00.000-07:00

Workflow in Alfresco WCM is a powerful tool to implement review & approval processes. At the same time, workflow development is not a trivial exercise. Alfresco uses JBoss jBPM as a foundation, with Alfresco javascript to define actions. The workflow itself is defined as and XML file, with mixed in the javascript. You can deploy one or more workflows as an AMP file into the Alfresco web archive.

There is a tool for workflow development called jBPM Designer. It's an Eclipse based plugin, which has some nice graphical features to define workflows. It's not required to use this tool. Alternatively you can take an sample workflow and alter it using a regular text editor. Ensure to change module name, module directory name, namespace and possibly variable names. Otherwise the workflow might clash if the same sample workflow has already been deployed.

I haven't quite figured out what's the most efficient way to develop workflows. Code changes happen in the workflow's task definition, and every time you update it, you need to have it somehow redeployed in Alfresco. If you make code changes directly in the deployed workflow, you can restart the server to have the changes take effect. Alfresco will automatically create a new version of the workflow definition. There may be a way to redeploy the updated workflow through Alfresco's workflow console, but I haven't seen any concrete examples of that yet. I believe this works easier when you use the jBPM Designer, because you're supposed to be able to redeploy straight from the IDE.

It would be fantastic if the development process could be streamlined, to a point that you don't have to write too much code. I'm thinking about some sort of ant build process, which asks a series of questions, and based on the questions, it will generate the necessary artifacts for you. This could really save a lot of time. I think such a wizard-like approach already exists in the jBPM process designer, so that's another option as well.

Ultimately it would be great if workflows could be designed by the power user, using the Alfresco web client. This could be a form based interface to define task nodes, transitions, conditions, etc. There are some common components that may be re-usable, such as code to send e-mail alerts to assignees. This could be easily abstracted as a function that can be turned on or off and configured with GUI controls.

The challenge with the last approach is the deployment of the workflow. Is Alfresco capable of (re-)deploying a workflow definitions after a workflow has been configured.

I Pad

2010-05-09T12:23:00.000-07:00

Since a couple of weeks, I've been playing with an iPad, which my daughter Lilly won on an raffle at her school :-) To be honest, I wouldn't have bought one. It's definitely not a device one could not live without. But it's cool, and I like the whole tablet concept thing.

The touch screen is nice, and I like how you can easily navigate. It's a very shallow UI, but rich nonetheless. The YouTube, Mail and Safari apps are used frequently. Also installed the Weather Channel app, MLB, BBC, Skype, Doodle Buddy and some simple games.

The weak point of the iPad I find to be the data input. The virtual keyboard is big enough, but it's hard to type fast with one hand as you're holding the device with the other hand. It would be really nice if it had built in voice recognition. Maybe in the second generation iPad. There may even be apps out there that already do this.

Project Handoff

2010-05-04T07:37:00.000-07:00

I've done now a couple of project as a software consultant, and one of the crucial part always seems to be the handoff / the delivery. It's something you don't always think about when you start a new project. But ultimately the goal is to make the client proficient with most every aspect of the new system. The IT team needs to know how to compile, configure, deploy and to troubleshoot. End users have to be become familiar with the user interface.

One obvious way of knowledge transfer is through documentation. It's helpful to create some notes during the development phase. Using a wiki this is typically not too time consuming, and it's easy to publish pages as PDF or Word. Always understand what level of detail is needed. Depending of the client team experience, the documentation may need to be more or less verbose.

Sometimes it's useful to walk the customer through the deployment process. I think of this as the best way of knowledge transfer. One time I had to do two eight hour web conference sessions with a client to, since I had no direct access to the system. It was painful, but it forced me to walk the customer through the process step-by-step. Even fixed a few small bugs along the way :-)

Performance Testing Learnings

2010-05-03T07:47:00.000-07:00

The last few weeks I've been performance testing a proof-of-concept application with good old JMeter. Here are some of my findings.

To me, this testing is still more of an art form than a exact science. There are simply too many parameters that can be changed, so tweaking and system tuning seems to be going hand in hand.

The main things you can change in between tests is the # of concurrent requests, and the target throughput (transactions per minute). It's important to monitor the the JMeter logs to ensure there are no time-outs occurring. If you make the sampler time-out settings to conservative, this will sooner or later happen as load increases. What exactly is conservative depends on the type of site and audience.

What's important to find out is what kind of maximum throughput can be accomplished with a minimum amount of time-outs. You have to do several test runs to find this out, each time increasing the load on the system. For each load level you will also want to measure the average and median response times.

As you're running JMeter tests, it's very useful to profile the server to see how CPU and memory resources are being utilized. In my case I used YourKit for profiling Tomcat 6. It's easy to setup, and it can show you when resources are getting over utilized, or even locking issues.

Especially locking issues, caused by concurrent access to synchronized code blocks, can slow things down. When you are fully loading the application, and CPU does not spike all the way to 100%, that could be an indication of locking issues.

Monitoring Java heap memory is important when doing a lot of caching, and in general to see if the application has memory leaks. Provided you have no such leaks, you still want to find out the memory footprint of your application, to determine what kind of memory should be allocated to the JVM. When using caching, it can be useful to understand the footprint on an object (type) level, so you know how the application scales under different concurrent loads.

As performance tests are executed, you undoubtedly will end up trying to tune the full application stack. The prototype I tested used eXist DB and EHCache, and was deployed on Tomcat and fronted by Apache. In each components there are configuration properties that can be tweaked.

There is always a bottleneck in the stack, and the challenge is to find it. There can be a combination of weak links. The next steps would be tune each weak component, and to see if certain links can be replaced, or even skipped. You may end up tweaking the overall system architecture, based on the findings of the system tuning effort.

Stress Testing Basics

2010-04-19T11:32:00.000-07:00

Every once in a while you have to stress test a newly cobbled together system, to determine how it performs under a lot of load. From my experience, I'm finding that you want to find out is something like this:

How many transactions per second can it support?
What is the average response times under different load levels (concurrent threads)?
At what load level start requests to time out frequently?

Before you can even start testing, you have to come up with some sort of test design. This requires you quantify some of your anticipated usage.

How many daily users expected?
How many clicks per user session?
What kind of queries expected in a user session?
Different user profiles?
What is my goal for average/median response time?

Answering these questions will give you some constraints to work with, and will drive the test design & execution. The goal is to make the design as realistic as possible.

Sometimes you'll find that you need to re-think your goals a bit. A single server can only handle so many transactions per time unit. For certain usage load you have to start thinking about clustering or sharding.

To test a web site with JMeter requiring Basic Authentication

2010-04-14T09:18:00.000-07:00

I'm currently stress testing a web application with JMeter. My test plan has a handful of HTTP samplers, but the application is protected by Apache with basic authentication.

To allow the test plan to authenticate with each request, you need to add an HTTP Header Manager configuration element to the Thread Group. In it you add one field named "Authorization".

The only thing left is to set the header value, which can be obtained by looking at the requests of a browser. In Firefox this can be done with the Live HTTP Headers Add-on.

Grails, Ivy and Maven

2010-02-25T08:17:00.001-08:00

As I'm diving deeper into Grails 1.2.1, I get to learn more about the dependency resolution mechanism. I came across the following scenario:

Grails project with dependency in local Maven repo
Project compiles and runs fine, but Maven project dependency has a bug
Let's fix the bug, and do a 'mvn clean install'.
Now, let's run Grails project again, after doing a 'grails clean'.
Hmmm, the bug is still there, even as I fixed it and re-installed in my local Maven repo???

Interesting enough, it appears that Grails actually retrieves my Maven project dependency from the Ivy cache. There is a crude way around that, by deleting the Ivy cache altogether (rm -r ~/.ivy2/cache), but the next time you build the Grails project, it again will cache the Maven defined project. This can be seen when the log level in BuildConfig.groovy is set to "debug" level:

grailsHome: no ivy file nor artifact found for org.rivetlogic#crafter;1.7.0-SNAPSHOT
don't use cache for org.rivetlogic#crafter;1.7.0-SNAPSHOT: changing=true

Very strange. Hopefully I will find out how to change this soon.

Grails 1.2.1 Loader Constraint Violation

2010-02-25T04:30:00.000-08:00

I came across this error after adding a dependency to my Grails project:

java.lang.LinkageError: loader constraint violation: loader (instance of ) previously initiated loading for a different type with name "org/xml/sax/SAXParseException"

This error was thrown every time I tried to do a 'grails run-app'. This seems to be a known issue, but I found very little information online about what it was caused by. One thing is clear: The Grails class loader gets unhappy when it finds the same class in different libraries.

It turns out this SAX class org.xml.sax.SAXParseException is embedded in a couple of jars (xml-apis and xmlParserAPIs) used in my Maven project. The way to solve it was to add an excludes directive to the grails-app/conf/BuildConfig.groovy file:

dependencies {
runtime('org.rivetlogic:crafter:1.7.0-SNAPSHOT') {
excludes 'xml-apis', 'xmlParserAPIs'
}
}

Alfresco 3.3 and further...

2010-02-23T08:56:00.000-08:00

Alfresco recently released a slideshow about Alfresco 3.3 and 4.0, and it gives a good overview of where the product is heading.

Overall it seems that the Alfresco is becoming a more integrated platform, which is both good and bad. It depends on how it all will be packaged. If we can turn on and off certain modules quickly, then things will be fine.

One of the big trends in Alfresco 3.3 and further is a tighter integration with Spring Surf, a web application framework for WCM based sites. A major feature will be the notion of In Context Editing, implemented using a browser-based JavaScript toolbar. I hope this toolbar will at least work well in Firefox and IE, although the latter is always a challenge.

Of course, you don't have to be married to Spring Surf to deliver your web site. You are supposed to be able to write Alfresco Web Editor extensions to make the toolbar work in other application frameworks.

Completely new in 3.3 will be the CMIS based Web Delivery Runtime, and it appears to be a File System Receiver on steriods. It is supposedly technology agnostic, but is highly extensible and pluggable. It seems to be working in conjunction with "Deployment Services", something that allows the "core Alfresco repository to be utilized in the delivery tear". That sounds to me as if you'll have more flexibility and control over deployment within the web client.

This all seems very nice, and I look forward to the new release. However, it all appears to be highly technical, and I'm sure that lots more configuration will have to be done in tedious XML files.

I only hope that Alfresco would pay some more attention to the web user interface. There is always lots that can be improved upon, and the GUI is never an exception.

A first look at Grails

2010-02-19T11:38:00.000-08:00

Since a week or two I've been playing around with Grails, which really is a nice web framework. It's the java equivalent of Ruby on Rails, with Groovy as its language. The Groovy syntax is more streamlined than java, and in a lot of ways more powerful and compact.

Grails, as of version 1.2, also comes with dependency management, similar to Maven, but so far I've found it to be a bit buggy. The Grails class loader has some issues with multiple copies and/or versions of the same libraries, or so it seems.

When you run an application in Grails, it uses a build in Tomcat plugin, which is nice. You can create a war from your project, but while developing the embedded Tomcat works just fine. It speeds up development.

Grails also comes with its own testing framework, for both unit and integration tests. Test skeleton classes are automatically created when you define a new domain, controller or service.

The test classes are written in Groovy language, as is any other class in a Grails project. You may actually mix plain old Java and Groovy, as long as the file extension is '.groovy'.

These are just a few first observations on Grails, but there is much more. The framework is all about convention over configuration, and spots many best practices featured in other web frameworks. It also does dependency injection, as we now from JBoss Seam, it has a templating & tagging system, an object-relational-mapper, and a validation mechanism among other things.

More to follow...

Alfresco indexes out of date

2010-01-07T07:15:00.000-08:00

I recently restored my Alfresco 3.1.1 Enterprise deployment entirely because of some configuration issue, and found out at startup that consequently the Lucene index had become out of date. I saw messages in the catalina.out such as:

WARN [node.index.FullIndexRecoveryComponent] The indexes are not synchronized with the database.
WARN [node.index.AVMFullIndexRecoveryComponent] Index for avm store xxx is out of date
...
ERROR: Indexes not found for X stores.

Turns out Alfresco won't start when it encounters this. In order to resolve I had to change a setting in WEB-INF/classes/alfresco/repository.properties, located in the exploded WAR. I set index.recovery.mode to AUTO. I tried FULL first, but that did not seem to help. Not sure as it was trying to recover in the background, but Alfresco did not start properly with FULL.

By default this setting is set to VALIDATE. I wonder why it would not be AUTO. As a description in the properties file it says: Validates and auto-recovers if validation fails. That seems to make a lot of sense for default behavior.

mv