HDInsight Hive Metastore fails when the database name has dashes or hyphens
Working in Azure HDInsight support today, we see a failure when trying to run a Hive query on a freshly created HDInsight cluster. Its brand new and fails on the first try, so what could be wrong?
Our Hive client app fails with this kind of error.
| Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:445)at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:619)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.hadoop.util.RunJar.run(RunJar.java:221)at org.apache.hadoop.util.RunJar.main(RunJar.java:136)Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientat org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1483)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2743)at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2762)at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:426)... 8 moreCaused by: java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:526)at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1481)... 13 moreCaused by: javax.jdo.JDOUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.NestedThrowables:org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:549)at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:499)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)at com.sun.proxy.$Proxy9.createDatabase(Unknown Source)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:578)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:598)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:436)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5509)at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:178)at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)... 18 moreCaused by: org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.at org.datanucleus.store.rdbms.valuegenerator.TableGenerator.createRepository(TableGenerator.java:261)at org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obtainGenerationBlock(AbstractRDBMSGenerator.java:162)at org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerationBlock(AbstractGenerator.java:197)at org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractGenerator.java:105)at org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGenerator(RDBMSStoreManager.java:2005)at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1386)at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3827)at org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.java:2571)at org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOStateManager.java:513)at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:232)at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNew(ExecutionContextImpl.java:1414)at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2218)at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)... 34 more |
We found out the problem is very simple. The Hive metastore database in HDInsight usually lives in an Azure SQL Database, and the name of that database can be customized. In this case it was a name "hive-meta-store" database name.
The error gives us a hint, but the error isn't clear about what caused the failure.
Caused by: org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it.
Apparently in HiveServer2 there is a bug where if the metastore database name is not always are escaped by tick marks 'hive-meta-store' or brackets [hive-meta-store] so instead, Hive gets an incorrect syntax error. On such clusters, the Hiveserver2 wouldn’t start. HiveServer2 wouldn’t start because of table “meta-store-database.dbo.SEQUENCE_TABLE” table
| 2016-01-16 02:14:10,228 DEBUG [main]: Datastore.Native (Log4JLogger.java:debug(58)) - SELECT NEXT_VAL FROM meta-store-database.SEQUENCE_TABLE WHERE SEQUENCE_NAME=<'org.apache.hadoop.hive.metastore.model.MDatabase'>2016-01-16 02:14:10,233 INFO [main]: DataNucleus.ValueGeneration (Log4JLogger.java:info(77)) - Error encountered allocating block of IDs : Couldnt obtain a new sequence (unique id) : Incorrect syntax near '-'. |
The known issue
The issue is being tracked here https://issues.apache.org/jira/browse/HIVE-6113 and in Hortonworks' internal Jira item HIVE-433.
Hortonworks is aware of the issue, and it already fixed in a later build as per reading HIVE-6113 has already upgraded the datanucleus version to 4.x. This upgrade will be available as part of Hive 2.0, which is not out yet in Azure HDInsight.
Until that is done, avoid dashes or hyphens in your custom metastore, else HiveServer2 will not start.