Java - write once, test everywhere

Yes it has been a while since my last post. Been sidetracked with various other distractions.

Referring to my opening blog, just yesterday I received a phone call for the very same Dr Andrew Arch of Melbourne. Last year I received a confidential internal email from where I was working mistakenly sent to me, rather than my namesake, which I had to reply that they had the wrong Andrew. Our worlds are colliding - lol.

I have a simple web application to make life simpler for myself and my clients who I'm currently contracting for on an ad-hoc basis that acts as a simple http file transfer portal. My clients cannot get out from behind their corporate firewall using ftp, as their ftp ports are blocked, so I have a simple website where we can upload and download files from, simply by using our browsers using standard http, allowing the exchange of files too big for email and avoiding having to burn CD's and courier them across to each other. No database is involved, the files are simply saved into a dedicated folder on the Linux box and retrieved on request. But even if they could technically ftp to my Linux server, these clients are not that tech savvy to put them (read me) through the angst of using an ftp client. Better to use technologies they are familiar with than introduce new ones if you want easy adoption and little hand holding. There are exceptions, but its a good heuristic. Even then keeping things as simple as possible and as much in their experience space as I can, I'm now being asked for a tutorial on how to login, select the file and open/save the selection from a list of hyperlinks! You can never underestimate the lack of technical knowledge/expertise/comfort in the general community.

I develop on Windows XP and run my website on a Ubuntu 8.4 linux server distro. Both my Windows desktop and the Ubuntu server are running Java 6. I have tested with Firefox, Opera, Safari, Lynx and IE 7.0 from both my Windows XP desktop and my Ubuntu Linux laptop. My clients use a mixture of Windows 2000 and Windows XP and run IE 6.0, constrained by their corporate SOE. This exposed two variations that I hadn't catered for that caused me some minor issues.

While my combination of platforms and browsers all passed with no problem, when the clients attempted to upload files, the file name was saved as the full UNC path. So I had to cater for that scenario, which was a pain. I couldn't adequately test for it in my environment, as I couldn't reproduce the behaviour, and I wasn't about to rebuild my machines to match their environment, especially as this web site is more of a gesture of good will on my part than a full fledged production solution warranting an exponential combinatorial explosion of test scenarios. Still, it was a scenario that I should have in all rights anticipated and allowed for, as it was in hindsight predictable. But coding a fix was really guess work without a way of reproducing the issue. It also raises the question of just how much testing is appropriate given the huge variations of browsers and OS's that a web site has to deal with? Particularly if you are a small startup with limited resources vs what such trivial issues do for your reputation.

The other one was a classic cross platform issue. On my Windows development platform, a file with multiple consecutive spaces in the file name gets saved to the dedicated file directory. To view what files are in this directory, a call to a backing bean returns a List of Files and a call to Java's File.getName() method displays the file name to the web browser, as the JSP iterates through all of the File objects in the List representing that dedicated file directory. Works on Windows with no problems, with the only idiosyncrasy being that the HTML collapses the multiple consecutive spaces down to a single space in the browser view, but the underlying reference to the file name remains unmodified. Now potentially, I could address this browser behaviour as well, encapsulating the file name in a <pre> tag to preserve the white space or replace all of the spaces with non breaking spaces, that is if it really bothered me that much.

On Linux the behaviour is a little different. The file on upload is saved correctly using the Java File object and an Output Stream, retaining the multiple spaces in the file name. However, the call to the Java File getName() method returns a file name that has had the multiple spaces collapsed, just like a browser does to consecutive white space characters. So now, when you request the file to download, it doesn't exist, because the file in the directory has multiple consecutive spaces in its name, and your requesting a file with only a single space in any one place within its name. So I am reluctantly resorting to collapsing any consecutive spaces from the file name before saving to disk, as I have no ability to influence the behaviour of the Java File object on Linux, as far as I'm aware.

A search on the Internet came up with nothing related, other than that it is not advisable to use spaces more so that just about any other character in a Linux file path, and even in Windows, I still run into various problems attempting to run various Java applications that have spaces in the path. The amusing thing is when the Java application installs itself into the Program Files directory, which obviously has a space in it, yet it cannot run because of that space. Eclipse and JBoss are two such examples. For some things, they run fine in the Program Files directory, for other things, they just crash and burn. So if it causes them so much trouble, why do they have their windows installers install into the Program Files directory? Madness I say.

So now saving files to my little file transfer portal runs the risk of modifying the file name, which is not a good thing. But alas, without having found an alternative solution, I have to tell my clients to use my little web file portal with caution, as I cannot guarantee the integrity of their file names. You have to love universal environments. If only they could all learn to live with a space or two.

character data type

Another little mapping issue between PostgreSQL and Hibernate. A varchar or character varying data type maps quite happily to a java String. However, a character data type does not. When attempting to do so you get something along the lines of:

javax.persistence.PersistenceException: org.hibernate.HibernateException: Wrong column type: char_column, expected: varchar(2)

To address this, you need to define the correct datatype for Hibernate to understand, and it is not "char" or "char(2)" or "character", as suggested in what references I could find, including Hibernate in Action. Instead the correct value is "bpchar". So if using annotations, it would look something like:

@Column(name = "char_column", length = 2, columnDefinition="bpchar")
@Length(min = 2, max = 2)
public String getCharColumn() {
return this.charColumn;
}

For a database schema definition that went something like:

char_column character(2) DEFAULT NULL::bpchar

Yet using direct jdbc, it works with no mapping issues at all, just like the uuid datatype does. Which raises the concern with JPA in general, and Hibernate in particular, how difficult is it to use these ORM frameworks and still have database agnostic, portable code? My code would not be coupled to the ORM mapping quirks per DB dialect if I just went completely with jdbc. Yet I was always under the impression that one of the benefits of these frameworks was that they were meant to hide much of the DB dialect differences from you and allow you to have more portable code. However, it would appear in reality to be just the reverse.

UUID and Hibernate - second update

After testing more scenarios, it appears that the previous solution posted is not complete. Foreign key and compound keys that use a uuid type will fail with what I have documented to date. In order for these to work, you need to take some additional steps.

In the same package as your entity classes containing uuid types, you will need to add a package-info.java class with the following entries:

@org.hibernate.annotations.TypeDefs(
{
@org.hibernate.annotations.TypeDef(
name="uuid",
typeClass = fully.qualified.path.CustomUUIDType.class)
}
)

package fully.qualified.path;

The annotations can also change, but do not need to:


@Id
@GeneratedValue(generator="my_custom_uuid")
@GenericGenerator(name="my_custom_uuid", strategy = "fully.qualified.path.CustomUUIDGenerator")
@Column(name = "id", unique = true, nullable = false, length = 36, columnDefinition="uuid")
@Type(type="uuid")
@NotNull
public String getId() {
return this.id;
}

public void setId(String id) {
this.id= id;
}

Or for a non-@Id field:

@Column(name = "field", unique = true, nullable = false, length = 36, columnDefinition="uuid")
@Type(type="uuid")
@NotNull
public String getFieldValue() {
return this.fieldValue;
}

public void setFieldValue(String fieldValue) {
this.fieldValue = fieldValue;
}

Additionally, you will also need to override the PostgreSQLDialect.java class with the following:

import java.sql.Types;

//This is to address the mapping exception when using uuid data type in PostgreSQL with Hibernate
//javax.persistence.PersistenceException: org.hibernate.MappingException: No Dialect mapping for JDBC type: 1111
//This makes the assumption that no other fields types will be using the java.sql.Types.OTHER data type
public class CustomUUIDPostgreSQLDialect extends org.hibernate.dialect.PostgreSQLDialect {

public CustomUUIDPostgreSQLDialect() {
super();
registerColumnType(Types.OTHER, "uuid");
registerHibernateType(Types.OTHER, "string");
}

}

Having done that, you will then need to update the persistence.xml file in your META-INF directory changing the following property:

<property name="hibernate.dialect" value="fully.qualified.path.CustomUUIDPostgreSQLDialect"/>

So for me at least, this is what I needed to do to get the postgreSQL uuid data type to work with Hibernate. If you are attempting to do the same, hopefully your journey will not be too different from what I have documented, and you will have smoother sailing than I have had.

UUID and Hibernate - update

I have finally worked out the annotations for using a custom UUID generator as per the previous post. The Hibernate documentation is not that clear on what is required, though it is in there. To quote:

"@org.hibernate.annotations.GenericGenerator allows you to define an Hibernate specific id generator. ...strategy is the short name of an Hibernate3 generator strategy or the fully qualified class name of an IdentifierGenerator implementation. You can add some parameters through the parameters attribute."

Now that may be clearer to you, than it was to me. And other than using some examples of their own custom generators as supplied by Hibernate that is it. And all the examples on the net lazily just use the same as what is in the Hibernate documentation.

Unfortunately, the Hibernate documented uuid example:

@Id @GeneratedValue(generator="system-uuid")
@GenericGenerator(name="system-uuid", strategy = "uuid")
public String getId() {

generates a 32 character unformatted hex string. You can also use the parameter attribute as suggested by the documentation to format it with a hyphen "-" separator, but as per my previous post, this returns an incorrectly formatted UUID that postgresql does not accept, which is the whole reason for this post in providing a solution to that shortcoming.

NB. this solution is dependent on the org.hibernate.annotations.GenericGenerator annotation and the org.hibernate.annotations.Type annotation, so it is not a pure EJB3 solution. The org.hibernate.validator.NotNull annotation is not really necessary in this case, especially as it duplicates the nullable = false attribute of the @column annotation.

The my_custom_uuid attribute value for the generator and the name attributes, can be anything that is meaningful to you, as long as they match in both the @GeneratedValue and the @GenericGenerator annotations.

An annotation example for the custom UUID generator is as follows:

@Id
@GeneratedValue(generator="my_custom_uuid")
@GenericGenerator(name="my_custom_uuid", strategy = "fully.qualified.path.CustomUUIDGenerator")
@Column(name = "id", unique = true, nullable = false, length = 36, columnDefinition="uuid")
@Type(type="fully.qualified.path.CustomUUIDType")
@NotNull
public String getId() {
return this.id;
}

public void setId(String id) {
this.id= id;
}

I probably should also mention here, as I overlooked doing so in the previous post, that there are differences in how to construct a Hibernate SessionFactory between supporting *.hbm.xml files and annotations. Ostensibly they can be mixed in the same project, but your mileage will probably vary in doing so. Personally, I have found some issues in attempting to have both methods coexist.

Creating a *.hbm.xml SessionFactory can be done as per the documented examples:

sessionFactory = new Configuration().configure().buildSessionFactory();

Additionally, within the *.hbm.xml file, the Entity classes are resource attributes of the mapping element, similar to the following:

<hibernate-configuration>
<session-factory name="PostgreSQLFactory">
<property name="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</property>
...
<mapping resource="relative/path/PlainEntity1.hbm.xml"/>
<mapping resource="relative/path/PlainEntity2.hbm.xml"/>
</session-factory>
</hibernate-configuration>

However, the annotated SessionFactory has some gotchas that again, are not really covered that well in the Hibernate documentation.

The Hibernate example has the following:

sessionFactory = new AnnotationConfiguration().buildSessionFactory();

But this is incorrect, and will not work. In my experience, you actually need to do something similar to the following:

sessionFactory = new AnnotationConfiguration().configure().buildSessionFactory();

Additionally, unless the session factory has information about the target database, such as the dialect and connection details in the hibernate.cfg.xml file, it will fail at runtime. Significantly, the examples in the Hibernate documentation do not include these values. So if you are following the Hibernate documentation, make sure that the configure() call can get these details from somewhere, even if they are missing from the examples. Another thing to be mindful of, is that the mapping attributes for *.hbm.xml files use a resource attribute, while for annotations, it uses class attributes.

<hibernate-configuration>
<session-factory name="PostgreSQLFactory">
<property name="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</property>
...
<mapping class="relative.path.AnnotatedEntity1"/>
<mapping class="relative.path.AnnotatedEntity2"/>
</session-factory>
</hibernate-configuration>

Because I have a number of different hibernate.cfg.xml files in my tests, I take an approach similar to the following:

File configFile = new File("relative_path/hibernate.cfg.xml");

AnnotationConfiguration config = new AnnotationConfiguration();
config.configure(configFile);
sessionFactory = config
.addPackage("fully.qualified.package")
.addAnnotatedClass(AnnotatedEntity1.class)
.addAnnotatedClass(AnnotatedEntity2.class)
.addAnnotatedResource(relative/path/PlainEntity1.hbm.xml)
.addAnnotatedResource(relative/path/PlainEntity2.hbm.xml)
.buildSessionFactory();

I have provided this example only so that you can see that there are a number of construction alternatives available that you can play with, to meet a variety of needs. Also be mindful that the same entity cannot have an entry in a hibernate.cfg.xml file as well as registered programmatically as per the above example.

Now I need to decide if I should bother spending any time on creating a Hibernate reverse engineering template to auto-create this correctly from a postgresql db. Hmmm, maybe not.

UUID and Hibernate

Before I begin, apologies for the poor formatting of this post. It is late and my few attempts to edit the HTML directly have not been too good, so I figured I would work out how to format correctly on subsequent posts, if I have the time.

I have been playing around with a postgresql DB, running v8.3 with the new uuid data type. This now means that most of the commonly used DB's support a uuid data type in various forms (SQL Server - uniqueidentifier, Oracle, MySQL, and PostgreSql - uuid, with DB2 yet to do so. Not sure on the support for the other DB engines).

Additionally, since Java 1.5, Java has supported the generation of UUID's with the java.util.UUID class.

However, the implementation (storage and interface) of UUID's differs from DB to DB, and the java.sql.Types has no GUID/UUID data type. When using JDBC, UUID strings work with no problems at all with the JDBC 4 postgresql driver and Java 6.

Unfortunately, Hibernate attempts to be too clever, and as the PostgreSql UUID datatype is unknown to it, its reverse engineering tools treats it as a Serializable oject. So on interaction with the database, it fails in three ways.

1. There is a datatype mismatch, which requires you to create a custom hibernate data type.
2. Hibernate treats the data as a bytea datatype, as it is Serializable, so you need to change the java datatype in the Entity object to String and the sql-type in the *.hbm.xml to "uuid".
3. The default uuid.hex generator in hibernate does not generate a correctly formatted uuid string. The default is with no hyphens. You can instruct the uuid generator to use hyphens, but when it does, it is in an incorrect custom built 8-8-4-8-4 format, instead of the strict 8-4-4-4-12 that postgresql requires. So you also need to create your own custom UUID generator.

Essentially, in the *.hbm.xml you would see something similar to the following:

<id name="nameOfField" type="FullyQualifiedPath.CustomUUIDType">
<column length="36" name="name_of_column" type="uuid"/>
<generator class="FullyQualifiedPath.CustomUUIDGenerator"/>


And in the Entity class, something similar to:

private String NameOfField;

public String getNameOfField() {
return this.NameOfField;
}

public void setNameOfField(String NameOfField) {
this.parent = NameOfField;
}

And naturally, a custom UUID Type that implements org.hibernate.usertype.UserType. The following is my implementation (which still requires a little work):

import java.io.Serializable;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Types;
import java.util.UUID;

import org.hibernate.HibernateException;
import org.hibernate.usertype.UserType;

public class CustomUUIDType implements UserType {

private static final int SQL_TYPE = Types.OTHER;
private static final String DB_OBJECT_TYPE = "uuid";

@Override
public Object assemble(Serializable cached, Object owner)
throws HibernateException {
//I don't fully understand what I am meant to do to honour this contract.
//So at present, just returning the cloned cached object as is.
//It is optional anyway.
return deepCopy(cached);
}

@Override
public Object deepCopy(Object value) throws HibernateException {
Object retValue = null;

if (value != null && value instanceof String) {
try {
//make sure that it is a correctly formatted value before returning it
UUID.fromString((String) value);
//as Strings are immutable, returning a new reference to the value is effectively cloning it
retValue = value;
} catch (IllegalArgumentException e) {
//return null if value is not correctly formatted,
//but potentially should return an Exception.
}
}
return retValue;
}

@Override
public Serializable disassemble(Object value) throws HibernateException {
//I don't fully understand what I am meant to do to honour this contract.
//So at present, just returning the cloned value object as is.
//It is optional anyway.
return (Serializable) deepCopy(value);
}

@Override
public boolean equals(Object x, Object y) throws HibernateException {
boolean eq = false;

if (x != null && y != null
&& x instanceof String && y instanceof String) {
try {
UUID lhs = UUID.fromString((String) x);
UUID rhs = UUID.fromString((String) y);

eq = lhs.equals(rhs);
} catch (IllegalArgumentException e) {}
}

return eq;
}

@Override
public int hashCode(Object x) throws HibernateException {
return (x != null ? x.hashCode() * 33 : 1234567890);
}

@Override
public boolean isMutable() {
return true;
}

@Override
public Object nullSafeGet(ResultSet rs, String[] names, Object owner)
throws HibernateException, SQLException {
Object retValue = null;

assert names.length == 1;

String uuidValue = rs.getString(names[0]);
if (rs.wasNull()) {
retValue = null;
} else {
try {
//make sure that it is a correctly formatted value before returning it
UUID.fromString(uuidValue);
retValue = uuidValue;
} catch (IllegalArgumentException e) {
//return null if data retrieved is not correctly formatted,
//but potentially should return an Exception.
}
}

return retValue;
}

@Override
public void nullSafeSet(PreparedStatement st, Object value, int index)
throws HibernateException, SQLException {
if (value == null) {
st.setNull(index, SQL_TYPE, DB_OBJECT_TYPE);
} else {
try {
//make sure that it is a correctly formatted value before returning it
UUID.fromString((String) value);
st.setObject(index, value, SQL_TYPE);
} catch (IllegalArgumentException e) {
//set null if data being set is not correctly formatted,
//but potentially should return an Exception.
st.setNull(index, SQL_TYPE, DB_OBJECT_TYPE);
}
}
}

@Override
public Object replace(Object original, Object target, Object owner)
throws HibernateException {
return original;
}

@Override
public Class returnedClass() {
return String.class;
}

@Override
public int[] sqlTypes() {
return new int[] {SQL_TYPE};
}
}

And of course the custom UUID Generator which implements the org.hibernate.id.IdentifierGenerator:

import java.io.Serializable;
import java.util.UUID;

import org.hibernate.HibernateException;
import org.hibernate.engine.SessionImplementor;
import org.hibernate.id.IdentifierGenerator;

public class CustomUUIDGenerator implements IdentifierGenerator {

@Override
public Serializable generate(SessionImplementor arg0, Object arg1)
throws HibernateException {
return UUID.randomUUID().toString();
}
}

And yes, I will discuss the whole issue of using surrogate keys, UUID's and collisions, etc, in another post.

So I have worked through all of this and got it working a treat with *hbm.xml files. But do you think I can do the same with annotations? It is just not working for me. I have everything working, but the custom UUID generator.

The following are the annotations that I'm using:

@Id
// @GeneratedValue(generator="system-uuid")
// @GenericGenerator(name="system-uuid", strategy = "uuid")
@GeneratedValue(generator="FullyQualifiedPath.CustomUUIDGenerator")
@Column(name = "id", unique = true, nullable = false, length = 36, columnDefinition="uuid")
@Type(type="FullyQualifiedPath.CustomUUIDType")
@NotNull
public String getId() {
return this.id;
}

public void setId(String id) {
this.id = id;
}

The custom type seems to be working fine, as I can retrieve the data from the DB, however, the UUID generator does not appear to be working at all. Hibernate is not even returning an error on an attempted insert. So that is as far as I have got at this stage in getting the generator working using annotations. Hopefully my next post will have the solution, but if any of my fantasised legions of readers were to have the solution or an idea of what I'm missing, feedback would be hugely appreciated. But my guess is that I will need to ask on the forums to find a solution...

In the beginning...

I've been prodded, and cajoled, and encouraged to write a blog for sometime by a small number of people, who for various reasons seem to think, rightly or wrongly, that I may make a positive contribution of some sort in doing so. So I have finally acquiesced and here is my blog, though I have no expectations of establishing a following, and at this stage I cannot say how regularly I will post.

It will not be a personal exposé, or contain emotional outpourings, other than perhaps the occasional frustrated rant. Instead, it will cover my attempt to launch a start up in a very amateurish fashion, and technical issues that I hit in doing so.

I guess before anything else, I should start out with a note about identity. Just in case anyone attributes any of my comments or opinions to someone else, and therefore misrepresenting them. Arch is not that common a surname, but on the web, there appears to be quite a number of Andrew Arch's. Internet searches over the last decade for Andrew Arch would always come up with a number of them, here in Australia and overseas. With three of us here in Melbourne Australia.

There's the Andrew Arch who published some CSIRO papers and who always came up in the initial search engine results some 10 years ago, but who now seems to have disappeared from the search engines and apparently is some distant cousin of mine. Then there is Dr Andrew Arch of W3C, recently of Vision Australia. I don't know if we are related in any way, but I have been assured that I'm related to all the Arch's who live in the Melbourne, if not the Victorian area. However, that is a bold claim. For the last few years, this Dr Andrew Arch was the only Andrew Arch that showed up in search engines, blowing away any previous footprint that either myself or the CSIRO Andrew Arch may have had, which can only be a good thing, at least for me. However, I have a sneaking suspicion that the Dr Andrew Arch and the CSIRO Andrew Arch are one and the same person. Names can be very confusing, as simply not enough of an identifier. I guess I could always just contact him and find out, but nah... Mysteries are sometimes best left as mysteries, because then you can construct all sorts of fanciful stories based on what you don't know.

So I do a search just now, to see how pervasive Dr Andrew Arch still is, and what do you know. I'm beginning to pop up in Google and Yahoo, now that I have a few public profiles about the place. So I guess, to capitalise on that, it is in my personal interests to start blogging as a means of building on that profile, and perhaps even benefiting a little from some mistaken identity with our esteemed Dr Andrew Arch. Spreading the footprint of the name Andrew Arch can only be mutually beneficial, surely?

So now, I just have to work out what I want to achieve out of this blog, if anything at all. Then, what to write and at what level to convey it. I guess it would help if I identified my target audience. That would be a good first step.