Terry Mueller

How to use the Liferay transaction manager with Spring MVC portlets

2012-03-07T09:30:47Z

Liferay provides a class-loader-proxy (CLP) for their Transaction Manager.

1. Add the following to your spring context.xml:

<context:component-scan base-package="foo.bar" />
<bean id="txManager"
  class="com.liferay.util.spring.transaction.TransactionManagerClp">
</bean>
<tx:annotation-driven transaction-manager="txManager" />

2. Add the following Spring annotations to your controller methods:

@Transactional(readOnly = true, propagation = Propagation.REQUIRED)

3. Add the following log4j to your portal-log4j-ext.xml (or log4j.properties) to see that the correct tx/session handling is being performed - once per request not once per service invocation.

<category name="org.springframework">
  <priority value="INFO" />
</category>

How to log portlet performance with Liferay

2011-11-22T23:28:02Z

I'm currently helping a client to ensure their Liferay Portal meets the performance targets set by the business. Since many of the portlets are custom-developed, we need to identify which portlets are causing high latencies under load.

Set the following logger in your Log4j configuration, in our case, tomcat/lib/META-INF/portal-log4j-ext.xml:

    <category name="com.liferay.portlet.InvokerPortletImpl">
       <priority value="DEBUG" />
    </category>

This will cause the performance of each portlet render to be logged, for example:

render for welcomeportlet_WAR_welcomeportlet takes 1 ms
render for employmentportlet_WAR_linksportlet takes 80 ms
Create instance cache wrapper for schoolbizportlet_WAR_schoolbizportlet_INSTANCE_BcM7
serveResource for latestnewsportlet_WAR_latestnewsportlet takes 2 ms
serveResource for latestnewsportlet_WAR_latestnewsportlet takes 2 ms
render for schoolbizportlet_WAR_schoolbizportlet_INSTANCE_BcM7 takes 1 ms
render for staffsearchportlet_WAR_staffsearchportlet takes 2 ms
render for welcomeportlet_WAR_welcomeportlet takes 4 ms
serveResource for latestnewsportlet_WAR_latestnewsportlet takes 2 ms
render for myessentialsportlet_WAR_myessentialsportlet takes 271 ms
render for linksapplicationsportlet_WAR_linksportlet takes 724 ms
render for myessentialsportlet_WAR_myessentialsportlet takes 795 ms
render for myessentialsportlet_WAR_myessentialsportlet takes 41 ms
serveResource for latestnewsportlet_WAR_latestnewsportlet takes 2 ms
serveResource for latestnewsportlet_WAR_latestnewsportlet takes 2 ms
serveResource for latestnewsportlet_WAR_latestnewsportlet takes 2 ms
serveResource for latestnewsportlet_WAR_latestnewsportlet takes 2 m

In this example, two of the portlets stands out as taking a lot more time (700ms+) than the others.

The attached PERL script calc_percentiles.pl will read a liferay log and generate a simple report on portlet performance.

Usage:

cat liferay*.log | perl calc_percentiles.pl

Example output:

Portlet N p0 p50 p80 p90 p99 p100 aupportlet_WAR_aupportlet 5565 0 1 1 1 2 677 contentmarketportlet_WAR_contentmarketportlet 11743 0 1 1 1 4 255 employmentportlet_WAR_linksportlet 11742 18 99 128 168 383 1686 errorportlet_WAR_errorportlet 5 0 0 0 0 0 0 latestnewsportlet_WAR_latestnewsportlet 11742 7 11 13 14 102 746 linksapplicationsportlet_WAR_linksportlet 1281 30 1216 1945 2693 3976 10128linksdepartmentresourcesportlet_WAR_linksportlet 386 19 50 71 90 349 406 mybookmarksportlet_WAR_myfavouritesportlet 1036 0 0 0 0 0 0 mydetailsportlet_WAR_mydetailsportlet 842 48 141 150 159 251 1303 myessentialsportlet_WAR_myessentialsportlet 13874 0 641 1247 1639 3090 8518 myfilesportlet_WAR_myfavouritesportlet 628 1 4 6 9 24 207 notificationsportlet_WAR_notificationsportlet 11743 2 4 5 6 18 1584 schoolbizportlet_WAR_schoolbizportlet 11742 0 0 0 0 0 0 staffsearchportlet_WAR_staffsearchportlet 11742 1 2 2 2 3 422 welcomeportlet_WAR_welcomeportlet 13057 0 1 7 9 34 290

How to run the unit tests for Liferay Portal trunk

2011-11-11T05:39:09Z

Running the unit tests for Liferay Portal with MacOS X is straight-forward once you know how.

Set up the environment

Install Ant 1.7+
Install MySQL 5.0+
Install Java 6.0+

Set environment variables:

export JAVA_OPTS="-Xmx512m -XX:MaxPermSize=256m"
export COMPUTERNAME=$HOSTNAME

Create the directory structure

The following commands:

mkdir liferay-trunk
cd liferay-trunk
svn co http://svn.liferay.com/repos/public/portal/trunk/ src
svn co http://svn.liferay.com/repos/public/plugins/trunk/ plugins

will create the directory structure:

liferay-trunk/
    src/
    plugins/

Set up the MySQL database for testing

Install/start MySQL 5.x (in my case, 5.5.14)
mysql -u root and run the following to create an empty database called lportal_trunk:
```
create database lportal_trunk character set utf8;
grant all on lportal_trunk.* to ''@'localhost';
```
(so username/password not required)

Create an empty legacy ZIP (not sure what this was intended for but it is needed):

cd liferay-trunk/src
cd sql
mkdir legacy
zip -r legacy.zip legacy
cd ..
echo "sql.legacy.zip=sql/legacy.zip" > test.$COMPUTERNAME.properties

Note: test.<username>.properties gets overridden each run!

Create test.$COMPUTERNAME.properties as follows:

minimal.suffix=
db.mysql.schema=lportal_trunk
mysql.executable=/usr/bin/mysql
mysql.arg.line=-u root -e "source create${minimal.suffix}/create${minimal.suffix}-mysql.sql"

Create release.$COMPUTERNAME.properties as follows:

lp.plugins.dir=${project.dir}/../plugins/build

The build environment is hard-coded for Windows, so create a dummy script called cmd.exe and it to your PATH:
```
#!/bin/sh
echo $*
```
Edit ./tools/db-upgrade/portal-ext.properties and ./portal-impl/test/portal-test.properties and change the database to lportal_trunk:
```
jdbc.default.url=jdbc:mysql://localhost/lportal_trunk?useUnicode=true&characterEncoding=UTF-8&useFastDateParsing=false
```
Enable the plugins script to be executed:
```
chmod +x plugins/ext/create.sh
```

Run the unit tests

Run the tests:

cd liferay-trunk/src
ant -buildfile build-dist.xml unzip-tomcat
cd ../plugins
ln -s ../bundles
cd ../src
ant tests

Review output for errors, for example:
(there does not appear to be any simple way to create an HTML report)

[terry@localhost portal]$ find . -name test-results -exec grep -r errors {} \; | cut -d" " -f2,3,5,6
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.jmx.MessageBusManagerTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.ValidatorTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.jmx.DestinationManagerTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceMonthlyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.IteratorDispatcherDestinationTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.ArrayDispatcherDestinationTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceDailyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceWeeklyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceYearlyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.jmx.DestinationStatisticsManagerTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.UnicodePropertiesTest" tests="2"
errors="0" failures="0" name="com.liferay.portal.kernel.util.StringUtilTest" tests="3"
errors="0" failures="0" name="com.liferay.util.VersionTest" tests="5"
errors="0" failures="0" name="com.liferay.portal.service.ServiceTestSuite" tests="24"
errors="0" failures="0" name="com.liferay.portal.util.MiscTestSuite" tests="5"
errors="0" failures="0" name="com.liferay.portal.service.persistence.PersistenceTestSuite" tests="880"

Script to delete Liferay users

2011-12-19T23:44:37Z

The following BeanShell script can be used to bulk delete all non-admin users in a Liferay instance, including their groups, subscriptions, announcements, assets, blogs, etc.:

import com.liferay.portal.util.PortalUtil;
import com.liferay.portal.service.UserLocalServiceUtil;
import java.util.concurrent.*;

pool = Executors.newFixedThreadPool(20);

for (user : UserLocalServiceUtil.getUsers(0, 99999)) {
  if (user.isDefaultUser() || PortalUtil.isOmniadmin(user.getUserId())) {
    System.out.println("Skipping user " + user.getScreenName());
  } else {
    final Object userToDelete = user;
    pool.execute(new Runnable() {
        public void run() {
            System.out.println("Deleting user " + userToDelete.getScreenName());
            UserLocalServiceUtil.deleteUser(userToDelete);
        }
    });        
  }
}

Paste this script into Control Panel > Server Administration > Scripts, with the "BeanShell" selected as the language.

Check the logs for the progress and/or errors.

Note: Use with caution! This script cannot be rolled back.

Improved JTop with CPU per interval

2011-11-08T04:57:43Z

The JTop plugin for jconsole provides information on the most resource-intensive Java threads.

Rejeev Divakaran has extended JTop to include CPU per interval instead of just the cumulative CPU. This makes JTop more useful for diagnosing transient spikes in CPU usage.

Do I have a Hibernate session leak in my Liferay portlets?

2011-11-01T04:48:30Z

Whilst diagnosing a performance problem and memory leak, I turned on the following logging for Liferay Portal 6.0 EE SP1 to get a better view of Hibernate:

log4j.logger.org.hibernate.SQL=DEBUG
log4j.logger.org.hibernate.jdbc=DEBUG
log4j.logger.org.hibernate.transaction=DEBUG

I started seeing this line which made me wonder if sessions were being leaked:

transaction completed on session with on_close connection release mode; be sure to close the session to release JDBC resources!

In the end, I enabled the following logging of Hibernate sessions - specifically to see the opening and closing of sessions:

log4j.logger.org.hibernate.impl.SessionImpl=TRACE

It turns out the above message is just a debug warning so it's nothing to worry about. The number of closes equals the number of opens:

[DEBUG] [http-9280-211] org.hibernate.impl.SessionImpl - opened session at timestamp: 5407217215401990
[TRACE] [http-9280-211] org.hibernate.impl.SessionImpl - setting flush mode to: MANUAL
[DEBUG] [http-9280-211] org.hibernate.jdbc.AbstractBatcher - about to open PreparedStatement (open PreparedStatements: 0, globally: 20)
[DEBUG] [http-9280-211] org.hibernate.jdbc.ConnectionManager - opening JDBC connection
[DEBUG] [http-9280-211] org.hibernate.SQL - select resourcepe0_.resourcePermissionId as resource1_38_, resourcepe0_.companyId as companyId38_, 
resourcepe0_.name as name38_, resourcepe0_.scope as scope38_, resourcepe0_.primKey as primKey38_, resourcepe0_.roleId as roleId38_, resourcepe0_.actionIds as actionIds38_ from 
ResourcePermission resourcepe0_ where (resourcepe0_.companyId=? )AND(resourcepe0_.name=? )AND(resourcepe0_.scope=? )AND(resourcepe0_.primKey=? )
[DEBUG] [http-9280-211] org.hibernate.jdbc.AbstractBatcher - about to open ResultSet (open ResultSets: 0, globally: 6)
[DEBUG] [http-9280-211] org.hibernate.jdbc.AbstractBatcher - about to close ResultSet (open ResultSets: 1, globally: 7)
[DEBUG] [http-9280-211] org.hibernate.jdbc.AbstractBatcher - about to close PreparedStatement (open PreparedStatements: 1, globally: 20)
[DEBUG] [http-9280-211] org.hibernate.jdbc.ConnectionManager - transaction completed on session with on_close connection release mode; be sure to close the session to release JDBC resources!
[DEBUG] [http-9280-211] org.hibernate.jdbc.ConnectionManager - releasing JDBC connection [ (open PreparedStatements: 0, globally: 14) (open ResultSets: 0, globally: 6)]
[TRACE] [http-9280-211] org.hibernate.impl.SessionImpl - after transaction completion
[TRACE] [http-9280-211] org.hibernate.impl.SessionImpl - closing session

Unable to open socket file: target process not responding or HotSpot VM not loaded

2011-10-31T07:24:39Z

When running Liferay Portal 6.0 EE SP1 + Tomcat 6.0.29, I found it impossible to take a heap dump to diagnose a memory leak. Firstly, jps wouldn't even list the process ID so I had to use ps instead:

$ jps
9745 Jps

Secondly, when creating the dump with jmap I always got the following error:

$ jmap -dump:file=dump.hprof 7642
7642: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

The fix turned out to be somewhat unexpected. I simply had to remove the explicit declaration of the java.io.tmpdir system property from our Tomcat start-up script, i.e.:

-Djava.io.tmpdir=%CATALINA_BASE%/temp

Now both jps and jmap work fine:

$ jps
4916 Jps
4068 WrapperSimpleApp


$ jmap -dump:file=dump.hprof 4068
Dumping heap to /foo/bar/dump.hprof ...
Heap dump file created

How to determine if a user is a member of a community using the Liferay API

2011-10-31T05:46:31Z

My client needed to determine whether a user is a member of a specific community from within a Liferay hook. The answer is very simple:

User user = ...;
Group group = GroupLocalServiceUtil.getGroup(PortalUtil.getDefaultCompanyId(), "My Community");

if (GroupLocalServiceUtil.hasUserGroup(userId, groupId)) {
   // I am a member!
}

How to use time zones with Liferay Portal

2011-06-14T13:34:58Z

Liferay Portal's handling of time zones isn't as straight forward as it might appear. But it is easy to configure once you know how.

Inform Liferay of the time zones:
- Create ROOT/WEB-INF/classes/system-ext.properties with the default time zone, for example:
```
    user.timezone=Australia/Perth
```
  where the value of user.timezone is a legal value supported by Java's TimeZone class.
- Create ROOT/WEB-INF/classes/portal-ext.properties with a comma-separated list of the time zones in use, for example:
```
    time.zones=Australia/Perth, Australia/Sydney, Australia/Adelaide
```
When Liferay starts for the first time, it creates a default user for the company. This user is assigned the time zone obtained from the value of user.timezone.

@see CompanyLocalServiceImpl#checkCompany(String webId, String mx, String shardName)

The data stored for the default user(s) can be viewed using:
```
select * from User_ where default = 1
```
The default user is not visible in the Control Panel but it is possible to edit the default user's time zone using Control Panel > Portal > Settings > Miscellaneous > Display Settings > Time Zone.
Whenever a new user is created, it is assigned the current time zone of the default user, irrespective of the current value of the user.timezone system property.

@see UserLocalServiceImpl#addUser()
If Liferay has been running for a while without a correct user.timezone system property, it is possible to update all current (and future) time zones using:
```
update User_ set timezoneid = 'Australia/Perth'
```

How to export ehcache statistics to a CSV file

2011-05-24T04:49:24Z

While running load testing and tuning the cache settings for Liferay, I need to periodically export the ehcache statistics to a CSV file. I had a quick look at jmxtrans but in the end, I wanted to write a JMX client myself as a learning exercise.

Here is the code:


package au.com.permeance.ehcache;

import java.net.MalformedURLException;
import java.util.HashMap;

import javax.management.Attribute;
import javax.management.MBeanServerConnection;
import javax.management.ObjectInstance;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

public class CacheStatisticsExporter {

    private static String[] ATTRIBUTE_NAMES = new String[] { "AssociatedCacheName", "CacheHits", "CacheMisses", "ObjectCount" };

    public static void main(String[] args) {
        if (args.length != 4 || !args[1].matches("[0-9]+")) {
            System.err.println("usage: java " + CacheStatisticsExporter.class.getName() + " host port username password");
            System.exit(1);
        }

        String host = args[0];
        int port = Integer.valueOf(args[1]);

        HashMap<String, Object> map = new HashMap<String, Object>();
        String[] credentials = new String[2];
        credentials[0] = args[2];
        credentials[1] = args[3];
        map.put("jmx.remote.credentials", credentials);

        // Output header
        for (String attributeName : ATTRIBUTE_NAMES) {
            System.out.print(attributeName);
            System.out.print(",");
        }
        System.out.println();

        // Output values
        try {
            JMXConnector c = JMXConnectorFactory.newJMXConnector(createConnectionURL(host, port), map);
            c.connect();
            MBeanServerConnection connection = c.getMBeanServerConnection();
            for (ObjectInstance instance : 
                connection.queryMBeans(null, new ObjectName("net.sf.ehcache:type=CacheStatistics,*"))) {
                for (Object attrObject : connection.getAttributes(instance.getObjectName(), ATTRIBUTE_NAMES)) {
                    Attribute attr = (Attribute) attrObject;
                    System.out.print("\"");
                    System.out.print(attr.getValue());
                    System.out.print("\",");
                }
                System.out.println();
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static JMXServiceURL createConnectionURL(String host, int port) throws MalformedURLException {
        return new JMXServiceURL("rmi", "", 0, "/jndi/rmi://" + host + ":" + port + "/jmxrmi");
    }

}

The code itself is pretty straight forward. The usage is:

java -classpath target/classes/ au.com.permeance.ehcache.CacheStatisticsExporter host port \
     username password > host-ehcache-stats-`date +%Y%m%d-%H%M`.csv

A fragment of sample output is:

AssociatedCacheName,CacheHits,CacheMisses,ObjectCount,
"com.liferay.portal.util.WebCachePool","0","0","0",
"com.liferay.portal.kernel.dao.orm.FinderCache.com.liferay.portlet.messageboards.model.impl.MBBanImpl.List","0","1","1",
"com.liferay.portal.kernel.dao.orm.EntityCache.com.liferay.portal.model.impl.LayoutImpl","162","30","17",
"com.liferay.portal.service.impl.PortletLocalServiceImpl","0","0","0",
"com.liferay.portal.util.JavaScriptBundleUtil","2","1","1",
"com.liferay.portal.kernel.dao.orm.FinderCache","0","0","0",
"com.liferay.portal.kernel.dao.orm.FinderCache.com.liferay.portal.model.impl.PasswordPolicyImpl","0","1","2",
"com.liferay.portal.freemarker.LiferayCacheStorage","0","0","0",
"com.liferay.portal.kernel.dao.orm.EntityCache.com.liferay.portal.model.impl.ContactImpl","35","5","4",
"com.liferay.portal.kernel.dao.orm.FinderCache.com.liferay.portal.model.impl.PluginSettingImpl","1","3","2",
"com.liferay.portal.kernel.dao.orm.FinderCache.com.liferay.portlet.social.model.impl.SocialActivityImpl","0","2","3",
"com.liferay.portal.kernel.dao.orm.FinderCache.Users_Roles","27","6","4",
"com.liferay.portal.kernel.dao.orm.FinderCache.com.liferay.portal.model.impl.CompanyImpl.List","1","1","1",
"com.liferay.portal.kernel.webcache.WebCachePool","0","0","0",
"com.liferay.portal.kernel.dao.orm.EntityCache.com.liferay.portlet.expando.model.impl.ExpandoTableImpl","9","0","1",
"com.liferay.portal.velocity.LiferayResourceCacheUtil","0","0","33",

After importing this into Calc, I could easily re-sort it identify those objects which had an unsatisfactory CacheMisses value.

How to use CSS selectors to hide Liferay web forms on submit

2011-05-05T08:07:02Z

div.portlet-content-container div div.portlet-msg-success + form{
display: none;
}

How to make navigation bars and breadcrumbs accessible

2010-09-07T01:14:24Z

Navigation bars and breadcrumbs are simply a collection of links. To make these accessible, wrap them in a <map>. Normally, a <map> is used for image maps but this is what is recommended by technique H50 of WCAG 2.0. I tested this with SortSite and our breadcrumbs now pass the accessibility tests.

Example:

<map title="Breadcrumbs">
  <a href="/">Home</a> 
  &gt; <a href="/foo">Foo</a> 
  &gt; <a href="/bar">Bar</a> 
  ...
</map>

In this example, the <map> does not affect the rendering of the breadcrumbs.

How to debug Liferay email

2010-08-20T09:19:59Z

To turn on logging of Liferay mail and JavaMail, add the following to your Log4j configuration in webapps/ROOT/WEB-INF/class/META-INF/portal-log4j-ext.xml after the last <appender> entry:

    <logger name="com.liferay.util.mail">
        <level value="DEBUG"/>
    </logger>

For setting up SMTP using gmail, add the following to webapps/ROOT/WEB-INF/classes/portal-ext.properties:

mail.session.mail.smtp.socketFactory.class=javax.net.ssl.SSLSocketFactory
mail.session.mail.smtp.host=smtp.gmail.com
mail.session.mail.smtp.port=465
mail.session.mail.smtp.auth=true
mail.session.mail.smtp.user=<your gmail email address>
mail.session.mail.smtp.password=<your password>

Internet access on the move

2010-05-11T16:28:38Z

Here is my speed test sitting a SouthWest Coach Lines bus. Not bad!

They've just put a Telstra NextG wireless modem in a cupboard at the back of the bus. No external antenna or anything.

They also have a USB charging plug for every seat but it does nothing for my iPhone.

Two CMIS projects at Apache?

2009-12-12T13:33:55Z

A second CMIS project has been proposed as an incubator project at Apache:

[PROPOSAL] OpenCMIS incubator for Content Mangement Interoperability Services (CMIS)

This has raised a lot of interest within the existing Chemistry project (as shown in the subsequent discussion above) since they are also an incubator project for CMIS!

Chemistry:

"Apache Chemistry is an effort to provide a Java (and possibly others, like JavaScript) implementation of the upcoming CMIS specification , consisting of a high-level API for developers wanting to manipulate documents, a low-level SPI close to the CMIS protocol for developers wanting to implement a client or a server, and default implementations for all of the above. Chemistry aims to cover both the AtomPub and SOAP bindings defined by the CMIS specifications."

OpenCMIS:

"OpenCMIS provides a Java implementation of the OASIS CMIS specification. This includes a library to connect as a consumer to a CMIS repository, and a library to provide the CMIS protocol handlers on top of an existing repository. All the protocol bindings defined by the CMIS specification will be supported...

As the APIs are pretty different right now, contributing the OpenCMIS code to Chemistry will be very hard to do - but on a mid-term perspective, we will review our options to merge OpenCMIS with Chemistry..."

How to install the Java Plugin on Firefox with Fedora 12 x86_64

2009-12-11T02:40:02Z

After much frustration, I finally stumbled on the correct answer about installing Java Plugin on x86_64 on Linux: plugindoc.mozdev.org/linux-amd64.html#java-sun. This shows that the correct library to link is called libnpjp2.so. I had tried libjavaplugin_jni.so which seemed to be the closest match to libjavaplugin_oji.so which everyone else spoke about. So, in short:

ln -s /usr/java/jdk1.6.0_16/jre/lib/amd64/libnpjp2.so ~/.mozilla/plugins/libnpjp2.so

Entering about:plugins in the address bar shows it is registered:

Open source isn't perfect but it's pretty good

2009-12-08T02:19:23Z

I've recently been working with Eclipse BPEL and Apache ODE and fairly quickly found bugs with both. These bugs were easy enough to fix if you have the inclinition to debug them. In all cases, I was impressed that after contributing the patch to the respective bug tracking tools, it wasn't long before they were tested, reviewed and committed!

Eclipse BPEL

Bug 296532 - NPE when process folder contains files without a file extension = 7 days
Bug 296538 - Publish to the server does not preserve sub-directories = 7 days

Apache ODE

ODE-725 - Patch to add support for postgres to dao-jpa-db = <1 day
ODE-731 - ODE console doesn't work in IE6 = <1 day
ODE-732 - Patch for java.lang.ArrayIndexOutOfBoundsException = <1 day

So if you find a bug in an open source product, please fix it and commit a patch!

How to secure Apache ODE using Liferay users and groups

2009-12-04T08:07:30Z

Rather than using a separate database (or file) of users and roles, we decided to re-use the security features of Liferay Portal. This follows on from How to secure Apache ODE which you should read first.

Note: The following steps were performed with Apache ODE 2.1 snapshot (trunk), Apache Tomcat 6.0.18, Liferay Portal CE 5.2.3 and PostgreSQL 8.4. They should work with some modifications for other configurations.

Firstly, we need two database views that comply with Tomcat's DataSourceRealm:


CREATE USER tomcat PASSWORD 'tomcat';
CREATE SCHEMA tomcat AUTHORIZATION tomcat;

GRANT USAGE ON SCHEMA lportal TO tomcat;     
GRANT REFERENCES ON TABLE lportal.user_, lportal.users_usergroups, lportal.usergroup TO tomcat; 
GRANT SELECT ON TABLE lportal.user_, lportal.users_usergroups, lportal.usergroup TO tomcat; 

CREATE VIEW tomcat.users AS
    SELECT screenname as user_name, password_ as user_pass FROM lportal.user_;

CREATE VIEW tomcat.user_roles AS
    SELECT u.screenname AS user_name, ug.name AS role_name
    FROM lportal.user_ u, lportal.users_usergroups uug, lportal.usergroup ug
    WHERE u.userid = uug.userid AND uug.usergroupid = ug.usergroupid;

ALTER TABLE tomcat.users OWNER TO tomcat;
ALTER TABLE tomcat.user_roles OWNER TO tomcat;

In this example for PostgreSQL, the Liferay tables are stored in the lportal schema and the new tomcat views are created in the tomcat schema. A user called tomcat is created to own these views.

Next, we add the DataSourceRealm to $TOMCAT/conf/server.xml, inside the <GlobalNamingResources> area:


    <Resource
        name="jdbc/tomcat"
        auth="Container"
        type="javax.sql.DataSource"
        maxActive="10"
        maxIdle="3"
        maxWait="10000"
        username="tomcat"
        password="tomcat"
        driverClassName="org.postgresql.Driver"
        url="jdbc:postgresql://localhost/ncp" />

Finally, we declare the Realm inside the <Context>, providing the names of the tables and columns:


    <Realm
        className="org.apache.catalina.realm.DataSourceRealm"
        debug="99"
        dataSourceName="jdbc/tomcat"
        roleNameCol="role_name"
        userCredCol="user_pass"
        userNameCol="user_name"
        userRoleTable="tomcat.user_roles"
        userTable="tomcat.users" />

In Liferay Portal, create a User Group called "ode-admin" and assign your users to this group.
Next, restart Tomcat.
Open the ODE console using http://localhost:8080/ode/.
When prompted, enter the username and password of a Liferay user in the ode-admin user group.

How to secure Apache ODE

2009-12-04T05:15:20Z

Apache ODE does not have any security when installed out-of-the-box. It is simple to use Tomcat security to protect the ODE admin console.

Add a user and role to $TOMCAT_HOME/conf/tomcat-users.xml:

<role rolename="ode-admin"/>
<user username="admin" password="xxxxxxxx" roles="ode-admin"/>

Declare the security constraints in ode/WEB-INF/web.xml:

<web-app>
...
    <security-role>
        <role-name>ode-admin</role-name>
    </security-role>

    <security-constraint>
        <display-name>Security constraint for the ODE admin console</display-name>
            <web-resource-collection>
            <url-pattern>*.html</url-pattern>
            <url-pattern>/processes/InstanceManagement</url-pattern>
            <url-pattern>/processes/ProcessManagement</url-pattern>
            <url-pattern>/processes/DeploymentService</url-pattern>
            <url-pattern>/deployment/*</url-pattern>
            <http-method>GET</http-method>
            <http-method>POST</http-method>
        </web-resource-collection>
        <auth-constraint>
            <role-name>ode-admin</role-name>
        </auth-constraint>
        <user-data-constraint>
            <transport-guarantee>NONE</transport-guarantee>
        </user-data-constraint>
    </security-constraint>

    <login-config>    
        <auth-method>BASIC</auth-method>
    </login-config>

</web-app>

Restart Tomcat.
Now, when interacting with the ODE console at http://localhost:8080/ode/, it will prompt for a username and password.

If you tail the access log, you can confirm that the SOAP admin calls are also protected. The processes themselves are not protected because we'll use WS-Security. In the following example, the remote user is "admin" if protected or "-" if not protected (see third column):

[terry@localhost logs]$ tail -f localhost_access_log.2009-12-04.txt 
127.0.0.1 - admin [04/Dec/2009:13:00:22 +0800] "POST /ode/processes/InstanceManagement HTTP/1.1" 200 731
127.0.0.1 - admin [04/Dec/2009:13:00:22 +0800] "POST /ode/processes/InstanceManagement HTTP/1.1" 200 731
127.0.0.1 - admin [04/Dec/2009:13:00:37 +0800] "POST /ode/processes/InstanceManagement HTTP/1.1" 200 731
127.0.0.1 - admin [04/Dec/2009:13:00:37 +0800] "POST /ode/processes/InstanceManagement HTTP/1.1" 200 731
127.0.0.1 - admin [04/Dec/2009:13:00:52 +0800] "POST /ode/processes/InstanceManagement HTTP/1.1" 200 731
...
127.0.0.1 - - [04/Dec/2009:13:13:23 +0800] "POST /ode/processes/HelloWorld HTTP/1.1" 200 255 -
...

How to run the unit tests of Liferay Portal 5.2.3 CE

2009-11-21T06:38:56Z

Running the unit tests for Liferay Portal with Linux is straight-forward once you know how.

Set up the environment

Install Ant 1.7+
Install MySQL 5.1+
Install Java 6.0+

Set environment variables:

export JAVA_OPTS=-Xmx512m -XX:MaxPermSize=256m
export COMPUTERNAME=$HOSTNAME

Create the directory structure

The following commands:

cd ~
mkdir liferay
wget http://downloads.sourceforge.net/project/lportal/Liferay%20Portal/\
5.2.3/liferay-portal-src-5.2.3.zip?use_mirror=transact
unzip liferay-portal-src-5.2.3.zip
mv liferay-portal-src-5.2.3 portal

will create the directory structure:

liferay/
    portal/

Set up the MySQL database for testing

Install/start MySQL 5.x (in my case, 5.1.39)

mysql -u root and run:

create database lportal character set utf8;
grant all on lportal.* to ''@'localhost';

(so username/password not required)

Create a legacy ZIP (not sure what this was intended for but it is needed):

cd ~/liferay/portal/sql
mkdir legacy
zip -r legacy.zip legacy
cd ..
echo "sql.legacy.zip=sql/legacy.zip" > test.$COMPUTERNAME.properties

Note: test.<username>.properties gets overridden each run!

If using Linux, create sql/test.$COMPUTERNAME.properties as follows:

minimal.suffix=
mysql.executable=/usr/bin/mysql
mysql.arg.line=-u root -e "source create${minimal.suffix}/create${minimal.suffix}-mysql.sql"

If not using subversion and Windoze, edit build-test.xml and comment out all <exec> of cmd.exe to stop it attempting to revert files using subversion.

Run the unit tests

cd ~/liferay/portal
ant test

Review output for errors, for example:
(there does not appear to be any simple way to create an HTML report)

[terry@localhost portal]$ find . -name test-results -exec grep -r errors {} \; | cut -d" " -f2,3,5,6
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.jmx.MessageBusManagerTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.ValidatorTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.jmx.DestinationManagerTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceMonthlyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.IteratorDispatcherDestinationTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.ArrayDispatcherDestinationTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceDailyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceWeeklyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.util.RecurrenceYearlyTest" tests="26"
errors="0" failures="0" name="com.liferay.portal.kernel.messaging.jmx.DestinationStatisticsManagerTest" tests="1"
errors="0" failures="0" name="com.liferay.portal.kernel.util.UnicodePropertiesTest" tests="2"
errors="0" failures="0" name="com.liferay.portal.kernel.util.StringUtilTest" tests="3"
errors="0" failures="0" name="com.liferay.util.VersionTest" tests="5"
errors="0" failures="0" name="com.liferay.portal.service.ServiceTestSuite" tests="24"
errors="0" failures="0" name="com.liferay.portal.util.MiscTestSuite" tests="5"
errors="0" failures="0" name="com.liferay.portal.service.persistence.PersistenceTestSuite" tests="880"