Tapestry Training -- From The Source

Let me help you get your team up to speed in Tapestry ... fast. Visit howardlewisship.com for details on training, mentoring and support!

Monday, June 23, 2008

Maven for dependencies, not building

I've been working a bit with Ivy but finding it doesn't quite fit my needs (or that I have a lot more to learn). What I've found with Ivy is that it has its way of doing things, and part of The One True Path is that you should create a repository just for your organization ... they really hate the idea of depending on the central Maven repository.

In any case, my need is to make it easy to build Tapestry or Tapestry demos from source without a lot of fuss. That's where I like Maven's dependency management, especially for the most common artifacts.

The other problem I've had with Ant is that it doesn't get Maven scopes perfectly, or Maven artifacts (such as source JARs) without a lot of redundant typing.

So, for the mean time, I'm back to Ant plus the Maven Ant tasks.

What I want is to create and manage four folders:

  • runtime
  • runtime-src
  • test
  • test-src

The main folders, runtime and test, contain JARs needed for compilation/execution of the production code, and for the test code. This could expand to include more of Maven's scopes in the future, such as provided, but it suits my current needs. The -src folders contain source JARs, which I find essential to being productive. Nothing irks me more than opening a base class or interface and getting that ugly view of just the method names because source isn't available. It gives me flashbacks to miserable sessions with WebLogic trying to figure out what the hell it was doing.

So, how can we get there without the rest of Maven? How about this:

<?xml version="1.0"?>
<project name="common" xmlns:mvn="urn:maven-artifact-ant">

    <!-- Names of common directories used in the build. -->

    <!-- Directory in which temporary files are created. -->
    <property name="target.dir" value="target"/>
    <property name="target.lib.dir" value="target/lib"/>

    <!-- Directory to which imported dependencies are placed. Four subfolders are created: runtime, runtime-src, test
         and test-src. -->
    <property name="lib.dir" value="lib"/>

    <path id="maven-ant-tasks.classpath" path="maven-ant-tasks-2.0.9.jar"/>

    <typedef resource="org/apache/maven/artifact/ant/antlib.xml" uri="urn:maven-artifact-ant"
             classpathref="maven-ant-tasks.classpath"/>


    <macrodef name="import-dependencies">
        <attribute name="scope"/>
        <attribute name="folder" default="@{scope}"/>
        <element name="dependencies" implicit="true"/>

        <!-- locals -->
        <attribute name="lib" default="${lib.dir}/@{folder}"/>
        <attribute name="lib-src" default="@{lib}-src"/>

        <attribute name="target-lib" default="${target.lib.dir}/@{folder}"/>
        <attribute name="target-lib-src" default="${target.lib.dir}/@{folder}-src"/>

        <attribute name="lib-fileset" default="@{scope}.dependency.fileset"/>
        <attribute name="src-fileset" default="@{scope}.dependency.sources.fileset"/>

        <sequential>
            <mvn:dependencies pomrefid="project.pom" verbose="true" usescope="@{scope}"
                              filesetid="@{lib-fileset}"
                              sourcesfilesetid="@{src-fileset}">
                <remoteRepository id="tapestry-nightly"
                                  url="http://tapestry.formos.com/maven-snapshot-repository">
                    <snapshots enabled="true"/>
                </remoteRepository>
                <remoteRepository id="maven-central"
                                  url=" http://repo1.maven.org/maven2"/>
                <remoterepository id="openqa-release"
                                  url="http://archiva.openqa.org/repository/releases/"/>
            </mvn:dependencies>

            <mkdir dir="@{target-lib}"/>
            <mkdir dir="@{target-lib-src}"/>
            <!-- Flatten them, so we can do a sync. -->
            <copy todir="@{target-lib}" flatten="true" preservelastmodified="true">
                <fileset refid="@{lib-fileset}"/>
            </copy>

            <copy todir="@{target-lib-src}" flatten="true" preservelastmodified="true">
                <fileset refid="@{src-fileset}"/>
            </copy>


        </sequential>
    </macrodef>

    <macrodef name="remove-overlap">
        <attribute name="source"/>
        <attribute name="target"/>

        <attribute name="fileset.property" default="@{source}.overlap.files"/>

        <sequential>
            <pathconvert pathsep="," property="@{fileset.property}">
                <fileset dir="@{source}"/>
                <flattenmapper/>
            </pathconvert>

            <delete dir="@{target}" includes="${@{fileset.property}}"/>
        </sequential>
    </macrodef>

    <target name="setup-libs" description="Copy dependencies to lib folder.">

        <mvn:pom file="pom.xml" id="project.pom"/>

        <!-- Delete the scratchpad space. -->
        <delete dir="${target.lib.dir}" quiet="true"/>

        <import-dependencies scope="runtime"/>

        <!-- For the moment, this is somewhat broken: test scope is a super-set of compile/runtime scope.
             Everything in the runtime folders will end up in the test folders, plus more. -->
        <import-dependencies scope="test"/>


        <!-- Snapshots come down with the datestamp/version number, we need to fix that. -->
        <move todir="${target.lib.dir}" preservelastmodified="true">
            <fileset dir="${target.lib.dir}"/>
            <!-- Turn the date/version stamp back into SNAPSHOT -->
            <regexpmapper from="^(.*)(\d{8}\.\d{6}\-\d+)(.*)$$" to="\1SNAPSHOT\3"/>
        </move>

        <!-- Delete the overlap between lib/runtime and lib/test -->

        <remove-overlap source="${target.lib.dir}/runtime" target="${target.lib.dir}/test"/>
        <remove-overlap source="${target.lib.dir}/runtime-src" target="${target.lib.dir}/test-src"/>

        <echo>*** Synchronizing ${lib.dir} ...</echo>
        <sync todir="${lib.dir}" verbose="true">
            <fileset dir="${target.lib.dir}"/>
        </sync>
        <echo>... done.</echo>

        <delete dir="${target.lib.dir}" quiet="true"/>
    </target>

</project>

Just add the Maven Ant tasks JAR and a pom.xml (that exists exclusively to identify dependencies) and you are off and running.

Some of my requirements are awkward to implement in Ant:

  • Flattened directories; Maven really wants to build a mirror of the repository, with directories for groups, but I want everything in a single directory.
  • Remove overlap; Maven's test scope includes the runtime scope, extra work is necessary to keep the test directory to just the additional JARs without redundantly including what's already in runtime
  • Timestamps; I want to sync the directories, not just copy in, to account for version number changes and I don't want to change timestamps unless a file has changed (to keep the IDE from wasting time rescanning and reparsing it)

The script creates a temporary directory under target, which is first used to flatten the copy from the local repository, then to remove the runtime vs. test redundancies before sync-ing it over the proper lib directory.

The best part of all this? Nothing happens until ant setup-libs rather than the frequent surprises that Maven gives you!

Eventually, I'll repackage some of this into a common build.xml that can be shared across modules. In the meantime, it's good enough for demos and labs in the workshop.

7 comments:

Massimo said...

Nice! Really nice!

Janick said...

On my current project (a webapp) we used to build with maven 2, but we did not manage to get a repeatable, stable build. About a year ago we also switched from maven 2 to Ant, and never looked back. We now tend to say we 'upgraded' to ant. Since that moment our build is repeatable, stable and there have been almost no changes to our build script.

We had a lot more trouble to maintain our maven build than we have now with ant. However, we did like the maven scopes, so we tried to keep that in our ant build. We organize our jars into 4 folders (our scopes): lib/ant, lib/compile, lib/deploy and lib/test. Only the jars in the deploy folder get packaged in the .war. These folders and jars are kept in version control.

We also find it essential to have the source of the jars that are used in the project. We just add the to the same directory. Since binary and source are there together, you won't forget to delete the sources if you remove a jar from your project nor forget to change the source if you upgrade a jar. If you apply a naming convention (we use a .zip extension) for the sources then ant can easily filter them out when creating a war.

This also works really well in IntelliJ: we attach these four directories as jar directories and source directories in the module configuration. When you add a jar or a source zip to one of the directories it gets automatically detected and indexed by IntelliJ, no further configuration required (same thing when you remove a jar or source zip).

Just to support you in your decision to use ant again.

Brian Fox said...

If you really want all the jars in one folder, just use dependency:copy-dependencies and it will put everything together for you.

Unknown said...

Howard,

I know you asked a few questions about Ivy and Maven 2 repository compatibility which may not have been answered as you like, but to clarify I don't think Ivy dictates its way. I use Ivy on some projects where I depend directly on Maven 2 repositories without any problem. Maven scopes are converted in configurations in Ivy, so you can use them without too much headaches (despite not as easily as with native m2).

The main problem I see ATM is dealing with sources and javadocs artifacts from Ant. I use IvyDE with eclipse so I have source and javadoc artifacts attachments with no problem, but outside this tool, I agree it's not easy to get source and javadoc artifacts from maven 2 repo using Ivy.

That being said, if all you need is maven 2 dependency management, with maven 2 repository and maven 2 philosophy for dependency management, I think using maven 2 ant tasks is the best option. Ivy is better suited for an environment where you need more flexibility and control over your dependency management, want to ensure build reproducibility and high control over conflict management.

Unknown said...

Xavier -- I'm looking forward to switching back to Ivy once it straightens out a few things (with non-code artifacts) and I have more time to check it out. I may be using it with Gant.

Brian -- Really missing the point. It would be nice if copy-dependencies was built into the Maven Ant Tasks, but I'm trying to avoid using Maven2 altogether, or requiring that it be installed.

Tim O'Brien said...

Howard, you are free to contribute to the Maven Ant tasks. When Brian pointed out the dependency:copy-dependencies goal, that was your cue to check out the project and contribute. We'll be waiting.

Unknown said...

Tim,

I've been fooled twice :-). I'm using the Maven Ant tasks for the moment, and probably something that wraps those and Ant inside a Groovy script in the future (there seems to be multiple options there). Repository: great. Transitive dependencies: great. Scopes for dependencies: great. Building with Maven: never again.