# Release Notes for Derby 10.4.1.3

> These notes describe the difference between Derby release 10.4.1.3 and the preceding release 10.3.2.1.

- Overview
- New Features
- Bug Fixes
- Issues
- Build Environment

## Overview

> Derby is a pure Java relational database engine using standard SQL and JDBC as its APIs.

> Derby functionality includes:

> - Embedded engine with JDBC drivers
> - Network Server
> - Network client JDBC drivers
> - Command line tools: ij (SQL scripting), dblook (schema dump) and sysinfo (system info)

## New Features

> This is a feature release. The following new features were added:

> - SQL ROW_NUMBER() window function, as described in the SQL 2003 standard section 6.10, for an empty, inlined window specification. The window clause, as described in section 7.11 is not implemented.
> - Unique constraints on nullable columns. SQL Feature T591.
> - Asynchronous replication with manual fail-over.
> - Table Functions. SQL Feature T326. This lets you run queries against any external data source which can be presented as a JDBC ResultSet. Table Functions are useful for data migration and integration. For more information, please consult the "Programming Derby-style table functions" section of the Derby Developer's Guide as well as the functional specification attached to DERBY-716.
> - Java Management Extensions (JMX) for Derby, allowing local and remote monitoring and management of a running Derby system (this release includes a basic set of MBeans - it is expected that more JMX functionality will be added in later releases).
> - SQL bracketed comments (/*..*/). SQL Feature T351.
> - ij continuation marker. Interactive ij shows > prompt until ; is entered.
> - A JDBC statement object cache in the client driver. Some limitations imposed by the presence of DERBY-3596.
> - Caching of the isolation level and the current schema in the client driver.
> - Implement a new multi-threaded buffer manager to get better scalability on machines with multiple CPUs or multiple cores.

## Bug Fixes

> The following issues are addressed by Derby release 10.4.1.3. These issues are not addressed in the preceding 10.3.2.1 release.

> Issue Id | Description
> DERBY-3603 | 'IN' clause ignores valid results, incorrect qualifier handling suspected
> DERBY-3589 | AllocPage.createPage() doesn't initialize minimumRecordSize correctly
> DERBY-3571 | LOB locators are not released if the LOB columns are not accessed by the client
> DERBY-3538 | NullPointerException during execution for query with LEFT OUTER JOIN whose inner table selects all constants.
> DERBY-3458 | dblook fails on TERRITORY_BASED databases
> DERBY-3442 | Reference Manual doesn't state limit on number of identity columns
> DERBY-3430 | Inconsistency in JDBC autogen APIs between Connection.prepareStatement(...) and Statement.execute(...)
> DERBY-3379 | "No Current connection" on PooledConnection.getConnection() if pooled connection is reused during connectionClosed processing
> DERBY-3373 | SQL "distinct" and "order by" needed together
> DERBY-3366 | Various formatting erros in L10N property files
> DERBY-3354 | Select from large lob table with embedded gives OutOfMemoryError
> DERBY-3352 | truncateTable crashed, Caused by: java.lang.NullPointerException
> DERBY-3350 | SQL CAST always marks its type as nullable even if the expression to be cast is not nullable
> DERBY-3347 | ERROR XSDB3: Container information cannot change once written
> DERBY-3343 | Subsequent calls to PreparedStatement cause SQLIntegrityConstraintViolationException on column that is "Generated always"
> DERBY-3322 | Server guide refers to phantom property in template policy file for the Network Server
> DERBY-3321 | NullPointerException for 'NOT EXISTS' with nested subquery
> DERBY-3316 | Leak in client if ResultSet not closed
> DERBY-3308 | Broken synchronization for event handling in ClientPooledConnection40
> DERBY-3303 | ArrayIndexOutOfBoundsException at MergeSort.compare
> DERBY-3302 | NullPointerException during recovery of database with territory-based collation
> DERBY-3301 | Incorrect result from query with nested EXIST
> DERBY-3299 | Uniqueness violation error (23505) occurs after dropping a PK constraint if there exists a foreign key on the same columns.
> DERBY-3298 | Getting Started manual needs clearer introduction to connection URL
> DERBY-3288 | wrong query result in presence of a unique index
> DERBY-3279 | Derby 10.3.X ignores ORDER BY DESC when target column has an index and is used in an OR clause or an IN list.
> DERBY-3278 | Developer's Guide topic cdevdvlp51654 has duplicate information on connection URL
> DERBY-3274 | Developer's Guide has duplicate information on database connections
> DERBY-3262 | Documentation is missing cross-references for connection URL attributes
> DERBY-3260 | NullPointerException caused by race condition in GenericActivationHolder
> DERBY-3257 | SELECT with HAVING clause containing OR conditional incorrectly return 1 row - should return 2 rows - works correctly with 10.2 DB
> DERBY-3253 | NullPointer Exception (NPE) from query with IN predicate containing two values and joining a view with a large table. ERROR 38000: The exception 'java.lang.NullPointerException' was thrown while evaluating an expression.
> DERBY-3247 | Activation for a dynamic ResultSet created from an Prepared/CallableStatement will not be closed until garbage collection indicates it is unused to the LCC and the LCC closes it
> DERBY-3244 | NullPointerException in ....B2IRowLocking3.searchLeftAndLockPreviousKey
> DERBY-3243 | (jdbc net client) exception during normal iteration through "ResultSet" of "select * from t"
> DERBY-3238 | When table contains large LOB values (> ~32K) trigger execution fails for that row with ERROR XCL30: An IOException was thrown when reading a 'BLOB'
> DERBY-3231 | Sorting on COUNT with OR and GROUP BY delivers wrong results.
> DERBY-3230 | Selecting data from a Table raises Error XN008: Query processing has been terminated due to an error on the server
> DERBY-3229 | testSysinfoLocale fails if derbyTools.jar is first in the classpath
> DERBY-3226 | BlobLocatorInputStream.read() and ClobLocatorInputStream.read() don't mask out sign bits
> DERBY-3221 | "java.sql.SQLException: The conglomerate (-5) requested does not exist." from Derby 10.3.1.4 embedded within Eclipse 3.3 and RAD 7.0
> DERBY-3215 | Potential NullPointerException in CachedPage class
> DERBY-3214 | Optimizer can see negative cost estimates when pulling Optimizables from the join order.
> DERBY-3198 | Using setQueryTimeout will leak sections
> DERBY-3194 | LOCALIZEDDISPLAY of CURRENT_TIMESTAMP returns only the TIME
> DERBY-3168 | Reference Manual lacks topics on trace-related connection URL attributes
> DERBY-3160 | SYSCS_GET_USER_ACCESS incorrectly treats the passed in user name as a SQL identifier and thus can reports the wrong user information
> DERBY-3094 | Grouping of expressions causes NullPointerException
> DERBY-3084 | CREATE SCHEMA in refman does not contain material on restrictions in sqlAuthorization mode
> DERBY-3079 | Database name is printed twice in derby.log on rollbacks when logStatementText is enabled
> DERBY-3060 | Network Server incorrectly assumes that all SQLExceptions with error code 08004 are caused by an authentication failure.
> DERBY-3044 | Typos in documentation
> DERBY-3023 | Different result rows depending on the sequence of INNER JOIN and OUTER JOIN
> DERBY-3001 | SYSTABLES documentation for TABLETYPE should include 'A' (Synonym)
> DERBY-2983 | The ResultSet returned by DatabaseMetaData.getFunctions() does not contain a required column named FUNCTION_TYPE.
> DERBY-2939 | Log file is flushed every time a log buffer gets full
> DERBY-2935 | DDMReader.readLengthAndCodePoint() decodes long integer incorrectly
> DERBY-2815 | ij doesn't start with J2ME / JSR169 / weme6.1 because attempting to find java.sql.Driver if ij.protocol property is specified
> DERBY-2733 | ij rolls through NullPointerException (NPE) with J2ME/JSR169/WEME 6.1.
> DERBY-2683 | tools and utilities guide: ij.URLCheck's list of checked attributes is not correct
> DERBY-2680 | Reference manual section "Setting attributes.." lacks upgrade=true attribute
> DERBY-2592 | Wrong description of IndexName field in public JavaDoc for LockTable
> DERBY-2585 | Mistaken description in page of CLOB
> DERBY-2559 | recreating a datasource using javax.naming.Reference from a ClientDataSource40 fails
> DERBY-2351 | ORDER BY with expression with distinct in the select list returns incorrect result
> DERBY-2270 | '18 ' (18 followed by four blanks) needs three more blanks
> DERBY-2142 | NullPointerException while using XAConnection/PooledConnection in a heavily contended multithreaded scenario
> DERBY-2128 | The word 'class' appears twice for the message SIF01.V
> DERBY-1823 | Derby Developer's Guide - Issues w/ User authentication and authorization extended examples section/paragraph
> DERBY-1585 | derbylang/procedureInTrigger: not able to create trigger due to an open ResultSet
> DERBY-1573 | Unsafe synchronization in NetworkServerControlImpl

## Issues

> Compared with the previous release (10.3.2.1), Derby release 10.4.1.3 introduces the following new features and incompatibilities. These merit your special attention.

> - Note for DERBY-3585: Shutting down the Network Server now supports user authentication, and in fact requires credentials when authentication is enabled.

> - Note for DERBY-3460: The two following reserved keywords are introduced: `NONE` and `CURRENT_ROLE`.

> - Note for DERBY-3301: Queries with nested EXIST, ANY or IN clauses now return correct results.

> - Note for DERBY-3026: The `frameworks` directory (and its contents) has been removed.

> - Note for DERBY-3013: The column default value can now also be specified as CURRENT_USER or SESSION_USER.

> - Note for DERBY-2351: An ORDER BY clause of a DISTINCT query which specifies to order by a column which was not in the DISTINCT list is now rejected, because the intent of the query is ambiguous. Previously, Derby instead produced non-distinct results. Also, an ORDER BY clause which specifies a table-name-qualified column alias is now rejected as invalid, where previously it was accepted.

> - Note for DERBY-2065: Error code changed for embedded connection when a connection with an open transaction is attempted closed.

> ### Note for DERBY-3585

> > #### Summary of Change

> > Shutting down the Network Server now supports user authentication, and in fact requires credentials when authentication is enabled.

> > #### Symptoms Seen by Applications Affected by Change

> > Previously, a network server running with user authentication didn't check for user credentials for server shutdown. Any client could shut down the server by calling NetworkServerControl with a shutdown command-line argument or by invoking the shutdown() method (provided the shutdown was initiated on the host running the server). While this generated a console warning (Connection refused : Invalid authentication.), the server shutdown proceeded and could also result in open databases not being properly closed.

> > Now, class NetworkServerControl supports user and password information as command-line and constructor arguments. When running a network server with user authentication, a server shutdown now requires user credentials; if the user authentication check fails, the client sees an authentication error and the running server remains intact. Note that Derby does not yet restrict the shutdown privilege to specific users: the server can be shut down by any user on the server machine who presents valid credentials.

> > In detail, to provide user credentials class org.apache.derby.drda.NetworkServerControl now supports new shutdown command-line options:

> > > - -user <username>
> > > - -password <password>

> > as well as two additional constructors with new parameters:

> > > - NetworkServerControl(String userName, String password)
> > > - NetworkServerControl(InetAddress address, int portNumber, String userName, String password)

> > These command-line options or constructor arguments must be used to enable a NetworkServerControl instance to shutdown a network server running with user authentication.

> > #### Incompatibilities with Previous Release

> > If running a network server without user authentication (the default) no command-line or API incompatibilities will be experienced.

> > However, some incompatibilities were introduced if running a network server with user authentication:

> > 1. The NetworkServerControl command-line usage changes for shutting down a sever:

> >    > java org.apache.derby.drda.NetworkServerControl shutdown

> >    now results in an exception:

> >    > 08004:Connection authentication failure occurred. Reason: Invalid authentication.

> >    The remedy is to provide credentials:

> >    > java org.apache.derby.drda.NetworkServerControl shutdown -user <username> -password <password>

> > 2. The NetworkServerControl API usage to programatically shutdown a server changes:

> >    > `NetworkServerControl nsc = new NetworkServerControl();
> >    > nsc.shutdown();
> >    > `

> >    results in a java.sql.SQLException:

> >    > Connection authentication failure occurred. Reason: Invalid authentication.

> >    The remedy is provide credentials to the NetworkServerControl constructor:

> >    > `NetworkServerControl nsc = new NetworkServerControl(user, password);
> >    > nsc.shutdown();
> >    > `

> > 3. Note that there is an edge case

> >    > `NetworkServerControl nsc = new NetworkServerControl();
> >    > nsc.start(console);
> >    > ...
> >    > nsc.shutdown();
> >    > `

> >    which currently fails with the SQLException shown above.

> >    A solution is to create the initial NetworkServerControl instance with user credentials:

> >    > `NetworkServerControl nscauth = new NetworkServerControl(user, password);
> >    > nscauth.start(console);
> >    > ...
> >    > nscauth.shutdown();
> >    > `

> >    Another option is to create a second NetworkServerControl instance with user credential arguments for the purpose of server shutdown:

> >    > `NetworkServerControl nsc = new NetworkServerControl();
> >    > nsc.start(console);
> >    > ...
> >    > NetworkServerControl nscauth = new NetworkServerControl(user, password);
> >    > nscauth.shutdown();
> >    > `

> > 4. If users have their own tests that use Derby's junit test framework, they'll have to use a test decorator that takes user credential arguments.

> > #### Rationale for Change

> > The previous behavior represented a security issue, because any client could shut down a network server running with user authentication from the same host without needing to provide user credentials.

> > #### Application Changes Required

> > Application code and scripts will need to be adjusted to provide user credentials for shutting down a network server that runs with user authentication.

> ### Note for DERBY-3460

> > #### Summary of Change

> > The two following reserved keywords are introduced: `NONE` and `CURRENT_ROLE`.

> > #### Symptoms Seen by Applications Affected by Change

> > If an application attempts to use one of these keywords, a syntax error (SQL state 42X01) is raised.

> > #### Incompatibilities with Previous Release

> > These two keywords were not previously reserved by Derby, and applications that rely on using them will break.

> > #### Rationale for Change

> > This change complies with the SQL standard and prepares Derby for supporting SQL roles in a future release.

> > #### Application Changes Required

> > Such user identifiers must be renamed so that they are not reserved keywords. See the Derby Reference Manual section titled "SQL Reserved Words".

> ### Note for DERBY-3301

> > #### Summary of Change

> > Queries with nested EXIST, ANY or IN clauses now return correct results.

> > #### Symptoms Seen by Applications Affected by Change

> > In the previous release, applications that executed SQL statements containing nested EXISTS, ANY or IN clauses could see fewer rows than those satisfying the query. In particular, rows that had the same value for one of the selected columns as another row might not have been returned.

> > #### Incompatibilities with Previous Release

> > None.

> > #### Rationale for Change

> > The previous behavior violated the ANSI SQL standard. The new behavior is correct.

> > #### Application Changes Required

> > Typically none, but applications must handle that the correct results are now returned.

> ### Note for DERBY-3026

> > #### Summary of Change

> > The `frameworks` directory (and its contents) has been removed.

> > #### Symptoms Seen by Applications Affected by Change

> > Users and applications that rely on any of the scripts or HTML files in the `frameworks` directory (which has been deprecated since the 10.2.1.6 release) or any of its subdirectories will no longer be able to find those files in the same location.

> > #### Incompatibilities with Previous Release

> > Applications or commands referencing files in the `frameworks` directory will fail.

> > #### Rationale for Change

> > In the 10.2.1.6 release, new and improved scripts were added in a new `bin` directory, intended to replace the scripts in the `frameworks` directory. The new scripts follow Apache conventions, and all scripts are located in a single directory, making them easier to find. Removing the old and deprecated scripts and the `frameworks` directory itself will eliminate a potential source of confusion and annoyance among users.

> > The `frameworks` directory has been deprecated since the 10.2.1.6 release, and has not been maintained since then. The 10.2.1.6 release notes announced the deprecation of the scripts in the `frameworks` directory, and an additional file (`frameworks.DEPRECATED.txt`) was added in the top-level directory of the 10.3.1.4 release, with the purpose of alerting users about this change. A warning message was also added to the scripts in the `frameworks` directory at the same time.

> > #### Application Changes Required

> > All references to the `frameworks` directory or its contents must be updated. The scripts in the `bin` directory may be used instead of the old scripts.

> ### Note for DERBY-3013

> > #### Summary of Change

> > The column default value can now also be specified as CURRENT_USER or SESSION_USER.

> > #### Symptoms Seen by Applications Affected by Change

> > None

> > #### Incompatibilities with Previous Release

> > None

> > #### Rationale for Change

> > Extend Derby's support for standard SQL constructions.

> > #### Application Changes Required

> > None.

> ### Note for DERBY-2351

> > #### Summary of Change

> > An ORDER BY clause of a DISTINCT query which specifies to order by a column which was not in the DISTINCT list is now rejected, because the intent of the query is ambiguous. Previously, Derby instead produced non-distinct results. Also, an ORDER BY clause which specifies a table-name-qualified column alias is now rejected as invalid, where previously it was accepted.

> > #### Symptoms Seen by Applications Affected by Change

> > ##### New rules for DISTINCT and ORDER BY

> > Applications which specify certain combinations of SELECT DISTINCT with ORDER BY will now receive an error message, whereas formerly such applications received non-distinct results.

> > As an example, take the following:

> > create table person (name varchar(10), age int);
> > insert into person values ('John', 10);
> > insert into person values ('John', 30);
> > insert into person values ('Mary', 20);
> > SELECT DISTINCT name FROM person ORDER BY age;

> > The query above is now rejected, with the error message:

> > > ERROR 42879: The ORDER BY clause may not contain column 'AGE', since the query specifies DISTINCT and that column does not appear in the query result.

> > If the AGE column is included in the DISTINCT list in the above query, there is no ambiguity

> > ##### New column alias rules

> > Applications which specify a column alias for a column in the SELECT statement, and which specify an ORDER BY clause which specifies that column alias qualified by the table name, will now receive an error indicating that the ORDER BY clause is invalid.

> > As an example, take the following:

> > create table t1 (i int, j int);
> > select t1.id as idcolumn1, t1.id as idcolumn2 from t1 order by t1.idcolumn1, t1.idcolumn2;

> > This query is now rejected, as there is no column named 'idcolumn1' in table 't1'. The error message is:

> > > ERROR 42X04: Column 'T1.IDCOLUMN1' is either not in any table in the FROM list or appears within a join specification and is outside the scope of the join specification or appears in a HAVING clause and is not in the GROUP BY list. If this is a CREATE or ALTER TABLE statement then 'T1.IDCOLUMN1' is not a column in the target table.

> > Valid forms of the query above are:

> > select t1.id as idcolumn1, t1.id as idcolumn2 from t1 order by idcolumn1, idcolumn2;

> > or

> > select t1.id as idcolumn1, t1.id as idcolumn2 from t1 order by t1.id, t1.id;

> > #### Rationale for Change

> > When the query ambiguously specifies both DISTINCT and ORDER BY, Derby was unsure whether to return the rows properly ordered, but non-distinct, or to return a distinct set of rows, but in an unknown order. Since no clear resolution of the ambiguity could be found, we chose instead to reject the query.

> > The rules for resolving column references in ORDER BY clauses have been enhanced to consider column aliases and column names more fully. Derby now uses different resolution rules depending on whether the ORDER BY column reference is table.column, or just column:

> > - if the table name is provided, we match against the underlying table name, and don't consider any aliases
> > - if the table name is NOT provided, we first match against the alias name, if present, and if no alias name matches then we match against the underlying source column name.

> > #### Application Changes Required

> > A query which specifies ordering by a non-distinct column should instead include the ORDER BY column in the DISTINCT list, to resolve the ambiguity about which values of that column should be used to distinctly identify the resulting rows.

> > A query which specifies table-name.alias-name should be rewritten to specify either simply alias-name, or table-name.column-name.

> ### Note for DERBY-2065

> > #### Summary of Change

> > Error code changed for embedded connection when a connection with an open transaction is attempted closed.

> > #### Symptoms Seen by Applications Affected by Change

> > In the previous release, calling Connection.close() on a connection with an open transaction raised an error with error code 25001 with the client driver, whereas the embedded driver raised error code 25000. The embedded driver has now been changed to raise the same error code as the client driver, i.e. 25001, as specified by the SQL standard.

> > #### Incompatibilities with Previous Release

> > Embedded applications that are dependent on the error code ever being "25000" could start failing. Embedded applications that are dependent on the error code never being "25001" could start failing.

> > #### Rationale for Change

> > Harmonize error codes raised by the client and embedded drivers, thereby also making the embedded driver compatible with the SQL standard.

> > #### Application Changes Required

> > Applications that are dependent on the error code must be changed to expect the new code.

## Build Environment

> Derby release 10.4.1.3 was built using the following environment:

> - Branch - Source code came from the 10.4 branch.
> - Machine - SunOS khepri32 5.10 Generic_127112-01 i86pc i386 i86pc
> - Ant - Apache Ant version 1.7.0 compiled on December 13 2006
> - JDK 1.4 - java version "1.4.2_05" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_05-b04) Java HotSpot(TM) Client VM (build 1.4.2_05-b04, mixed mode)
> - Java 6 - java version "1.6.0_04" Java(TM) SE Runtime Environment (build 1.6.0_04-b06) Java HotSpot(TM) Server VM (build 10.0-b18, mixed mode)
> - OSGi - The felix.jar was used to build org.apache.derby.osgi.EmbeddedActivator.
> - Compiler - The 1.6.0_04 javac was used to compile all classes.
> - JSR 169 - J2ME support was built using java.sun.com/j2me (cdc-1_1-fr-ri, jdbc_cdc1.0).