The Rabbit Hole

What starts out as a simple question (where do I put my JDBC drivers in Tomcat?) quickly leads down a rabbit hole…

Actually, this is a straightforward configuration topic - and not the true source of the rabbit hole. That starts with an unusually opinionated paragraph found in this page of the official Tomcat documentation (emphasis mine):

java.sql.DriverManager supports the service provider mechanism. This feature is that all the available JDBC drivers that announce themselves by providing a META-INF/services/java.sql.Driver file are automatically discovered, loaded and registered, relieving you from the need to load the database driver explicitly before you create a JDBC connection. However, the implementation is fundamentally broken in all Java versions for a servlet container environment. The problem is that java.sql.DriverManager will scan for the drivers only once.

So, on the one hand, for my simple Java web app being hosted on Tomcat, there are some straightforward JDBC set-ups which are well understood and commonly used. But on the other hand, there is clearly something more going on here… and down the hole we go.

Some JDBC History

In the olden days, when you wanted to use a JDBC driver to connect to your relational database from a Java application, you needed to explicitly load the relevant JDBC Driver class.

I’ll use MySQL as my example:

This is from a very old version of MySQL - version 5.1.5 to be specific. Do not use this in your modern code!

1
2
3
4
5
6
7
8

private void connectUsingDriverManagerTheOldWay() throws SQLException, ClassNotFoundException {
    Class.forName("com.mysql.jdbc.Driver"); // legacy class name! DO NOT USE!
    try ( Connection conn = DriverManager.getConnection(mySqlUrl, user, pass)) {
        final Statement statement = conn.createStatement();
        final ResultSet resultSet = statement.executeQuery(selectSql);
        showResults(resultSet);
    }
}

In the above fragment, the relevant line is:

1

Class.forName("com.mysql.jdbc.Driver"); // legacy class name! DO NOT USE!

The Class.forName() call is not directly related to JDBC - it is just a general purpose Java method for explicitly instructing the current classloader to load a class object from its fully-qualified name.

This is explained in the Java DDBC Tutorial as follows:

In previous versions of JDBC, to obtain a connection, you first had to initialize your JDBC driver by calling the method Class.forName.

and:

Any JDC 4.0 drivers that are found in your class path are automatically loaded. (However, you must manually load any drivers prior to JDBC 4.0 with the method Class.forName.)

Why was Class.forName Ever Needed?

But why did we ever need to use Class.forName? We always had our JDBC driver (the JAR file) on the classpath. Why could its classes not be accessed just like any other of our classes in core Java or in libraries in JAR files?

The specific technical reason is eloquently explained in this Stack Overflow answer from Joachim Sauer:

If you’re not using a current JDK (or if you have a JDBC driver that does not have the appropriate files set up to use that mechanism) then the driver needs to be registered with the DriverManager using registerDriver. That method is usually called from the static initializer block of the actual driver class, which gets triggered when the class is first loaded, so issuing the Class.forName() ensures that the driver registers itself (if it wasn’t already done).

That answer also makes the points that you need (a) a “type 4” JDBC driver which supports the automatic loading mechanism; and (b) a recent enough version of Java in which the loading mechanism is actually provided (but this has been available since Java 6).

Classpaths

Another way to answer this “why the need…?” question is to consider that none of the code in the above example makes direct reference to the specific vendor JDBC driver. It’s all handled via Java’s java.sql.* classes.

1
2
3
4
5

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

There is nothing in that code which would cause the Java runtime to try to load the required MySQL JDBC driver class - hence the need for an explicit line of code to force the issue.

There are different classpaths. The compile-time classpath contains all libraries and dependencies needed to compile the source code. We can see that there is nothing in our source code for connectUsingDriverManagerTheOldWay() which directly depends on the MySQL driver.

It is only at runtime (when we attempt to connect to the MySQL database) that we need the MySQL driver available - on the runtime classpath.

Type 4 Drivers

Returning to “type 4” drivers…

A “JDBC 4.0” driver is a type of JDBC driver which is implemented in pure Java. It is the most recently introduced “type” of JDBC driver - and all mainstream DBMSs almost certainly provide a type 4 driver, these days.

If you are interested in reading about other types, a good overview is provided by the Wikipedia page. In summary:

• Type 1 driver: JDBC-ODBC bridge
• Type 2 driver: Native-API driver
• Type 3 driver: Network-Protocol (middleware) driver
• Type 4 driver: Database-Protocol driver/Thin Driver (Pure Java)

How do you know if your driver is a Type 4 driver?

The simplest way is to locate the driver JAR file and look inside it (e.g. with a tool such as 7-Zip). There should be a META-INF directory containing a MANIFEST file. For my above example, the related manifest file was as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.6.5
Created-By: 1.4.2-b28 (Sun Microsystems Inc.)
Built-By: mysqldev

Name: common
Specification-Title: JDBC
Specification-Version: 4.0
Specification-Vendor: Sun Microsystems Inc.
Implementation-Title: MySQL Connector/J
Implementation-Version: 5.1.5
Implementation-Vendor-Id: com.mysql
Implementation-Vendor: MySQL AB

There we can see the following:

Specification-Title: JDBC
Specification-Version: 4.0

SPI and Automatic Driver Loading

Coming back to the following sentence from the Java JDBC tutorial:

Any JDC 4.0 drivers that are found in your class path are automatically loaded

How does that automatic loading work?

Side note: It’s not actually guaranteed that a Type 4 JDBC driver will be automatically loaded: there are some very early type 4 driver versions which may not have implemented this feature.

For example, the MySQL example I referenced above was from MySQL 5.1.5 - and the automatic loading mechanism was not supported. But by version 5.1.6, it was supported.

It is worth adding that MySQL version 5.1.x was first released in November of 2008 - so this is (in technology terms) bordering on ancient history. These are not versions you should be using today.

Back to automatic loading…

Automatic loading is provided via the Java SPI - the Service Provider Interface. The ServiceLoader documentation provides an overview. This is a general purpose mechanism used not only by JDBC but also by other areas such as JNDI, JAXP and so on.

A JDBC driver typically implements support for SPI by providing a file in META-INF/services named java.sql.Driver.

The contents of that file will vary from provider to provider. For example, for the 5.1.6 version of MySQL’s JDBC driver, the contents of that file are:

1

com.mysql.jdbc.Driver

That is to say, it’s the same value as the one we needed to use in our Class.forName example earlier.

Modern MySQL JDBC drivers now use the following: com.mysql.cj.jdbc.Driver The earlier example (com.mysql.jdbc.Driver) is from a legacy MySQL JDBC driver.

Automatic Loading Steps

We can now take a closer look at how loading happens…

To illustrate the auto-loading process I will assume the following:

a) a simple JDBC connection to a MySQL database using DriverManager:

1
2
3
4
5
6
7

private void connectUsingDriverManagerMySql() throws SQLException {
    try ( Connection conn = DriverManager.getConnection(mySqlUrl, user, pass)) {
        final Statement statement = conn.createStatement();
        final ResultSet resultSet = statement.executeQuery(selectSql);
        showResults(resultSet);
    }
}

b) The following JAR files included on the classpath, shown as Maven dependencies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

<dependencies>        
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>8.0.27</version>
    </dependency>
    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <version>1.4.200</version>
    </dependency>
</dependencies>

Why two drivers for two different databases? Because this will illustrate an important point about how automatic loading works.

The steps for my scenario are as follows:

1. The DriverManager.getConnection method is invoked from my code. The calling class (i.e. the class containing my code) is retrieved by the driver manager. This class is used in the next step, along with my connection URL, user ID and password.

From the DriverManager JavaDoc:

When the method getConnection is called, the DriverManager will attempt to locate a suitable driver from amongst those loaded at initialization and those loaded explicitly using the same class loader as the current application.

Note also that the

2. The driver manager calls a getConnection worker method which obtains the classloader for the calling class, using caller.getClassLoader().

From the DriverManager JavaDoc again:

The drivers loaded and available to an application will depend on the thread context class loader of the thread that triggers driver initialization

Classloaders in Java are a rabbit-hole unto themselves - and I don’t propose to go too far into this topic here. However, it is crucially important to how automatic loading works, because it relates to which resources (in our case which JDBC drivers) can be located, in different runtime configurations.

3. The ensureDriversInitialized method is called. This only happens once for our application - another critical point with important consequences. The DriverManager class uses a boolean driversInitialized field to track whether initialization has already happened, for any subsequent DriverManager.getConnection calls.

4. The ensureDriversInitialized method calls the ServiceLoader class to perform driver class loading:

1
2
3
4
5

// If the driver is packaged as a Service Provider, load it.
// Get all the drivers through the classloader
// exposed as a java.sql.Driver.class service.
ServiceLoader<Driver> loadedDrivers = ServiceLoader.load(Driver.class);
Iterator<Driver> driversIterator = loadedDrivers.iterator();

5. The service loader’s load method handles scanning the runtime classpath for resources (JAR files) which meet the required criteria - which in our case, is (a) the existence of a file named META-INF/services/java.sql.Driver, which (b) must contain the fully qualified name of the JDBC driver class to be loaded by the service loader. These are “registered” - which simply means their classes are added to a list of available JDBC driver classes (registeredDrivers) in the driver manager.

6. Specifically, ServiceLoader.load(Driver.class) will cause a new instance of the specific JDBC driver to be created. Typically, JDBC drivers will contain a static initialization block which is executed when the driver is created.

Here is the MySQL JDBC driver example (simplified slightly for this article):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

package com.mysql.cj.jdbc;

import java.sql.SQLException;

//
// This is a SIMPLIFIED version of the code
// for illustration purposes only!
//

public class Driver implements java.sql.Driver {
    //
    // Register ourselves with the DriverManager
    //
    static {
        try {
            java.sql.DriverManager.registerDriver(new Driver());
        } catch (SQLException E) {
            throw new RuntimeException("Can't register driver!");
        }
    }

}

7. The above code calls back into the driver manager and causes the driver to be added to the list of registered drivers:

1
2

// List of registered JDBC drivers
private final static CopyOnWriteArrayList<DriverInfo> registeredDrivers = new CopyOnWriteArrayList<>();

and:

1

registeredDrivers.addIfAbsent(new DriverInfo(driver, null));

8. Once all drivers have been registered, the driver manager sets driversInitialized to true. For our set-up, this means we end up with two registered drivers: The MySQL driver and the H2 driver. So, even though we were not even attempting to perform any H2-related database access, that driver is still registered at this point in the process.

9. The driver manager’s getConnection method then walks through the list of registeredDrivers, and invokes each one’s connect method:

1

Connection con = aDriver.driver.connect(url, info);

10. The first successful connection from the list is returned to the client application.

Some points in summary:

• This process means that we do not need any explicit references to specific driver implementations. The only database-specific details we need are the string properties needed for the URL, user ID and password.

• Drivers are loaded only once for my application. That happens “lazily”, the first time a DriverManager.getConnection statement is encountered. The DriverManager class cannot be instantiated - it uses private DriverManager(){}. Amongst other things, it acts as a holder for static fields, as we saw for the registeredDrivers list. These static fields will not be garbage collected until the class loader which loaded the DriverManager is itself eligible to be garbage collected (typically at the end of the program, or when a web application is unloaded). Therefore the list of registered drivers will remain available - and driversInitialized will remain true.

• If my application has more than one JDBC driver (as shown in the pom.xml example at the beginning of this section), then it’s possible that multiple connection attempts could be made, before one actually works. However, well-behaved drivers will typically perform a lightweight sanity-check on the provided URL, before attempting a more expensive connection.

• This process is the reason why modern JDBC drivers no longer require you to explicitly use Class.forName in your JDBC code. As we shall see later on, however, there can be exceptions where Class.forName may still be needed.

You could choose to sidestep the SPI process entirely by replacing the DriverManager with code such as the following - note the reference to the MySQL implementation class com.mysql.cj.jdbc.Driver:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// STRONGLY DISCOURAGED - requires explicit reference to a
// JDBC driver implementation class:
private void connectUsingExplicitDriver() throws SQLException {
    java.sql.Driver driver = new com.mysql.cj.jdbc.Driver();
    java.util.Properties connectionProps = new java.util.Properties();
    connectionProps.put("user", user);
    connectionProps.put("password", pass);
    try ( Connection conn = driver.connect(mySqlUrl, connectionProps)) {
        final Statement statement = conn.createStatement();
        final ResultSet resultSet = statement.executeQuery(selectSql);
        showResults(resultSet);
    }
}

This is discouraged because now you have a hard-coded reference to a vendor-specific class object: com.mysql.cj.jdbc.Driver.

Using DataSource instead of DriverManager

In the DriverManager JavaDoc it states:

The javax.sql.DataSource interface, provides another way to connect to a data source.

And:

The use of a DataSource object is the preferred means of connecting to a data source.

What’s that all about? Have we been doing it wrong up to now?

Here is my earlier MySQL example, rewritten to use a DataSource:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

private void connectUsingDataSourceMySql() throws SQLException {
    MysqlDataSource ds = new MysqlDataSource();
    ds.setURL(mySqlUrl);
    ds.setUser(user);
    ds.setPassword(pass);
    try ( Connection conn = ds.getConnection()) {
        final Statement statement = conn.createStatement();
        final ResultSet resultSet = statement.executeQuery(selectSql);
        showResults(resultSet);
    }
}

What are the advantages of using DataSource? Well, in my naive example, nothing, really. In fact, it requires me to import a specific implementation class:

1

import com.mysql.cj.jdbc.MysqlDataSource;

So, in that sense, it’s worse than the original DataSource version.

But if you look at the official Java JDBC tutorial for data sources, you will see the following advantages mentioned:

• DataSource objects can provide connection pooling and distributed transactions.
• Programmers no longer have to hard code the driver name or JDBC URL in their applications, which makes them more portable.
• DataSource properties make maintaining code much simpler.

The last two points in particular are more relevant to applications running in containers - such as web apps in a Tomcat application server. And it could be argued that “much simpler” is not always the case. What may be simpler fro you as a developer, may be more work for the web server administrator. The work is really just moved somewhere else.

(I will take a closer look at connection pooling next - but distributed transactions will not be discussed here.)

Basic Connection Pooling

I will use HikariCP. Here is the Maven dependency:

1
2
3
4
5

<dependency>
    <groupId>com.zaxxer</groupId>
    <artifactId>HikariCP</artifactId>
    <version>5.0.0</version>
</dependency>

Hikari support both the “old school” DriverManager approach using their jdbcUrl parmeter, and the “preferred” DataSource approach, using their dataSourceClassName parameter - although, as they take pains to point out:

We recommended using dataSourceClassName instead of jdbcUrl, but either is acceptable. We’ll say that again, either is acceptable.

To begin with I will use a MySQL pool using jdbcUrl. The simple reason for this is that currently, there is a known issue using the MySQL data source approach:

The MySQL DataSource is known to be broken with respect to network timeout support. Use jdbcUrl configuration instead.

It looks like an issue regarding this was opened in the MySQL bug tracker:

Incorrect implementation of Connection.setNetworkTimeout()

The issue was discussed from 2015 through 2017, but appears to have languished ever since.

Here is a very simple implementation of a HikariCP pool, using an enum to give us a singleton:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import java.sql.Connection;
import java.sql.SQLException;

public enum PoolDemo {

    INST;

    private final HikariConfig config = new HikariConfig();
    private final HikariDataSource ds;

    private final String mySqlUrl = "jdbc:mysql://localhost:3306/test";
    private final String user = "test_user";
    private final String pass = "xxx";

    PoolDemo() {
        config.setJdbcUrl(mySqlUrl);
        config.setUsername(user);
        config.setPassword(pass);
        ds = new HikariDataSource(config);
    }

    public Connection getConnection() throws SQLException {
        return ds.getConnection();
    }
}

Here is how it can be used:

1
2
3
4
5
6
7

private void connectUsingHikariPool() throws SQLException {
    try ( Connection conn = PoolDemo.INST.getConnection()) {
        final Statement statement = conn.createStatement();
        final ResultSet resultSet = statement.executeQuery(selectSql);
        showResults(resultSet);
    }
}

In this case, it is the execution of new HikariDataSource(config) which triggers the same DriverManager auto-registration and loading process as previously outlined. In other words, Hikari takes care of the JDBC driver loading.

This:

Connection conn = ds.getConnection()

has become this:

Connection conn = PoolDemo.INST.getConnection()

To use Hikari with a data source class name, instead of a connection URL, you can do this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

import com.zaxxer.hikari.HikariDataSource;
import java.sql.Connection;
import java.sql.SQLException;

public enum PoolDemo {

    INST;

    private final HikariDataSource ds = new HikariDataSource();

    //
    // WARNING - this is NOT RECOMMENDED for MySQL due to the issue
    // noted earlier relating to MySQL network timeout handling. But
    // it shows the approach you can use for other databases.
    //
    private final String dsClassName = "com.mysql.cj.jdbc.MysqlDataSource";
    private final String database = "test";
    private final String user = "test_user";
    private final String pass = "test_user123";

    PoolDemoB() {
        ds.setDataSourceClassName(dsClassName);
        ds.setUsername(user);
        ds.setPassword(pass);
        ds.addDataSourceProperty("databaseName", database);
    }

    public Connection getConnection() throws SQLException {
        return ds.getConnection();
    }
}

Tomcat with a Servlet

At long last, we can take a look at JDBC using Tomcat 10.0 (in this case, with a very simple JSP page and a servlet). The page displays a “success”/“fail” message depending on whether a database connection was made.

The servlet code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

import java.io.IOException;
import jakarta.servlet.ServletException;
import jakarta.servlet.annotation.WebServlet;
import jakarta.servlet.http.HttpServlet;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.logging.Level;
import java.util.logging.Logger;

@WebServlet(urlPatterns = "/demo_one")
public class DemoServletOne extends HttpServlet {

    private final String mySqlUrl = "jdbc:mysql://localhost:3306/test";
    private final String user = "test_user";
    private final String pass = "test_user123";
    private final String selectSql = "select id, name from test_table";

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {
        request.getRequestDispatcher("/WEB-INF/demo.jsp").forward(request, response);
    }

    @Override
    protected void doPost(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {

        String result = "failed to connect to DB";
        try {
            result = connectUsingDriverManagerMySql();
        } catch (SQLException ex) {
            Logger.getLogger(DemoServletOne.class.getName()).log(Level.SEVERE, null, ex);
        }

        request.setAttribute("result", result);
        request.getRequestDispatcher("/WEB-INF/demo.jsp").forward(request, response);
    }

    private String connectUsingDriverManagerMySql() throws SQLException {
        String result;
        try ( Connection conn = DriverManager.getConnection(mySqlUrl, user, pass)) {
            result = "successfully connected to DB";
            final Statement statement = conn.createStatement();
            final ResultSet resultSet = statement.executeQuery(selectSql);
            showResults(resultSet);
        }
        return result;
    }

    private void showResults(ResultSet rs) throws SQLException {
        System.out.println("---");
        while (rs.next()) {
            int id = rs.getInt("id");
            String name = rs.getString("name");
            System.out.println(String.format("%s: %s", id, name));
        }
    }
}

The JSP:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

<%@page contentType="text/html" pageEncoding="UTF-8"%>
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <title>Demo One</title>
    </head>
    <body>
        <h1>JDBC connection test</h1>

        <form action="demo_one" method="post">
            <p>
                <input type="submit">
                <span class="success">${result}</span>
            </p>
        </form>

    </body>
</html>

The Maven dependencies and build configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.me</groupId>
    <artifactId>JdbcResearchTomcat</artifactId>
    <version>1.0</version>
    <packaging>war</packaging>

    <name>JdbcResearchTomcat</name>

    <properties>
        <maven.compiler.source>16</maven.compiler.source>
        <maven.compiler.target>16</maven.compiler.target>
        <failOnMissingWebXml>false</failOnMissingWebXml>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>jakarta.platform</groupId>
            <artifactId>jakarta.jakartaee-web-api</artifactId>
            <version>9.1.0</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.27</version>
            <scope>provided</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>16</source>
                    <target>16</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>3.3.2</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                </configuration>
            </plugin>
        </plugins>
        <finalName>JdbcResearch</finalName>
    </build>

</project>

I build a WAR file using the above code and Maven pom.xml, and manually deploy it to my Tomcat’s webapps directory.

Driver Manager with a Local JAR

Our standard DriverManager.getConnection() code is used.

The MySQL JDBC driver is placed in the webapp’s WEB-INF/lib folder. I use the following Maven runtime scope for this:

1
2
3
4
5
6

<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>8.0.27</version>
    <scope>runtime</scope>
</dependency>

This gives the following runtime exception:

1

java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/test

Why does this fail? Why can’t my webapp find the JDBC driver bundled with the webapp?

This is because the driver manager has already run, as a part of Tomcat’s startup process. This is the point made by the Tomcat documentation I referenced at the start of this article.

It means that only libraries visible to the common class loader and its parents will be scanned for database drivers.

Therefore driversInitialized is already true and ensureDriversInitialized() does not run.

Why does Tomcat do this?

Memory Leaks in Servlet Containers

This requires another digression - and I will use the following presentation notes from Mark Thomas to help explain:

Diagnosing and Fixing Memory Leaks in Web Applications: Tips from the Front Line

A class is uniquely identified by:

  • Its name

  • The class loader that loaded it

Hence, you can have a class with the same name loaded multiple times in a single JVM, each in a different class loader. Web containers use this for isolating web applications. Each web application gets its own class loader.

An object retains a reference to the class it is an instance of. A class retains a reference to the class loader that loaded it. The class loader retains a reference to every class it loaded.

Retaining a reference to a single object from a web application pins every class loaded by the web application in the JVM’s memory.

These references often remain after a web application reload. With each reload, more classes get pinned in memory and eventually it fills up.

Note: The original presentation refers to the “Permanent Generation” memory area in the JVM. This has been replaced in more recent versions of the Oracle JVM with the Metaspace memory region - but the point about memory leaks still holds true.

So, one of the fundamental requirements of a servlet container (to manage multiple different web applications simultaneously) is in conflict with the way the DriverManager class handles automated driver loading.

Tomcat's Leak Prevention Listener

This is why Tomcat triggers the driver scan during Tomcat startup (via a configuration option which can be changed). It move this (and other similar core Java singletons) from each web application to the Tomcat platform, where they will not proliferate as web apps are re-loaded. But the consequence of this is:

Drivers packaged in web applications (in WEB-INF/lib) and in the shared class loader (where configured) will not be visible and will not be loaded automatically.

Instead, drivers can be placed in $CATALINA_HOME/lib, or, if it is provided, in $CATALINA_BASE/lib. These will be visible to the Tomcat classloader used to trigger driver loading.

More details can be found in the JreMemoryLeakPreventionListener documentation. See specifically the driverManagerProtection attribute.

Class.forName() - Still Needed

There are further consequences arising from this approach. For example, if you have different web applications which may rely on different versions of a JDBC driver, then placing these different driver JARs in the Tomcat lib directory may not work, since it is not guaranteed that each application will use the correct version of the driver.

In this case, you may still need to bundle each JDBC driver in the WEB-INF/lib directory of the related web application - and use Class.forName() to load it.

Tomcat with a Servlet and JNDI

Another approach is to take advantage of the previously discussed DataSource approach - this time via JNDI.

In this case, a JDBC connection can be defined via configuration, instead of in your web application’s code. The typical approach is to create this configuration in the Tomcat conf/context.xml file, along with a resource reference in your application’s WEB-INF/web.xml file.

Examples:

The context.xml entry:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

<Resource name="jdbc/MyTestDB"
    auth="Container"
    type="javax.sql.DataSource"
    username="my_username"
    password="my_password"
    maxTotal="10"
    maxIdle="2"
    maxWaitMillis="10000"
    driverClassName="com.mysql.cj.jdbc.Driver"
    url="jdbc:mysql://localhost:3306/my_db"/>

The web.xml entry:

1
2
3
4
5

<resource-ref>
    <res-ref-name>jdbc/MyTestDB</res-ref-name>
    <res-type>javax.sql.DataSource</res-type>
    <res-auth>Container</res-auth>
</resource-ref>

My web application can then access this data source as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import java.util.logging.Level;
import java.util.logging.Logger;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.sql.DataSource;

public enum MyAppDataSource {

    INSTANCE;

    MyAppDataSource() {
        try {
            Context initialContext = new InitialContext();
            Context context = (Context) initialContext.lookup("java:comp/env");
            ds = (DataSource) context.lookup("jdbc/MyTestDB");
        } catch (NamingException ex) {
            Logger.getLogger(MyAppDataSource.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    private DataSource ds;

    public DataSource getDataSource() {
        return ds;
    }

}

Just to note: This still needs to follow the same guidelines as described above regarding the correct location for the driver manager JAR file. It still relies on automatic driver registration.

Conclusions

There is nothing new in any of the above discussions, but I found much of the information somewhat scattered around different documents, presentations and discussions. Having a more detailed walkthrough has helped me gain a better understanding of the “why” behind the “how” regarding where to place my JDBC driver files.