Hadoop - création-Hive Metastore Database Error Tables - Stack Overflow ~ Stackoverflow Blog

I'm running through the Cloudera Manager (free edition) and I reached the point where the wizard is creating the Hive Metastore Database. This error is shown and halts the configuration process.

using /var/run/cloudera-scm-agent/process/40-hive-metastore-create-tables/hadoop-conf as HADOOP_CONF_DIR

I cant seem to find any information that might cause this?

Every thing has been configured correctly up to this point, everything installed and user names and passwords are correct.

Has anybody seen this error before? Thoughts?

Error Log:

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:688)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1094)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2337)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2370)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2154)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at com.cloudera.enterprise.dbutil.SqlRunner.open(SqlRunner.java:109)
at com.cloudera.enterprise.dbutil.SqlRunner.runSingleQuery(SqlRunner.java:80)
at com.cloudera.cmf.service.hive.HiveMetastoreDbUtil.countTables(HiveMetastoreDbUtil.java:191)
... 2 more

Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2540)
at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:612)
... 20 more

ok cloudera is using version HIVE 0.10, that doesn't support remote login

but you need to go around that bug, logging to the server that is getting the error the cloudera manager will tell you the ip

1) create login in to the Server that fails to install HIVE 2) Create a $HADDOP_HOME

$HADOOP_HOME="/usr/lib/hadoop/"

3) INSTALL postgres in the server that fails

`$ sudo apt-get install postgresql`

$ cat /etc/postgresql/9.1/main/postgresql.conf | grep -e listen -e standard_conforming_strings

modify this to lines in the file

listen_addresses = '*'
standard_conforming_strings = off

You also need to configure authentication for your network in pg_hba.conf. You need to make sure that the PostgreSQL user that you will create in the next step will have access to the server from a remote host. To do this, add a new line into pg_hba.con that has the following information:

host    <database>         <user>         <network address>         <mask>               password

Start PostgreSQL Server

$ sudo service postgresql start

Use chkconfig utility to ensure that your PostgreSQL server will start at a boot time:

chkconfig postgresql on

You can use the chkconfig utility to verify that PostgreSQL server will be started at boot time, for example:

chkconfig --list postgresql

Step 2: Install the Postgres JDBC Driver

Before you can run the Hive metastore with a remote PostgreSQL database, you must configure a JDBC driver to the remote PostgreSQL database, set up the initial database schema, and configure the PostgreSQL user account for the Hive user.

To install the PostgreSQL JDBC Driver on a Debian/Ubuntu system:

Install libpostgresql-jdbc-java and symbolically link the file into the /usr/lib/hive/lib/ directory.

$ sudo apt-get install libpostgresql-jdbc-java
$ ln -s /usr/share/java/postgresql-jdbc4.jar /usr/lib/hive/lib/postgresql-jdbc4.jar

Step 3: Create the metastore database and user account

bash# sudo –u postgres psql
bash$ psql
postgres=# CREATE USER hiveuser WITH PASSWORD 'mypassword';
postgres=# CREATE DATABASE metastore;
postgres=# \c metastore;
You are now connected to database 'metastore'.
postgres=# \i /usr/lib/hive/scripts/metastore/upgrade/postgres/hive-schema-0.10.0.postgres.sql
SET
SET
...

Now you need to grant permission for all metastore tables to user hiveuser. PostgreSQL does not have statements to grant the permissions for all tables at once; you'll need to grant the permissions one table at a time. You could automate the task with the following SQL script:

bash# sudo –u postgres psql
metastore=#         \o /tmp/grant-privs
metastore=#         SELECT 'GRANT SELECT,INSERT,UPDATE,DELETE ON "'  || schemaname || '"."' || tablename || '" TO hiveuser ;'
metastore-#           FROM pg_tables
metastore-#           WHERE tableowner = CURRENT_USER and schemaname = 'public';
metastore=#         \o
metastore=#         \i /tmp/grant-privs

You can verify the connection from the machine where you'll be running the metastore service as follows:

psql –h myhost –U hiveuser –d metastore
metastore=#

Step 4: Configure the Metastore Service to Communicate with the PostgreSQL Database

change the IP of the AWS amazon master Server, or your master server, don't use DNS name

$find / -name hive-site.xml
$nano /run/cloudera-scm-agent/process/27-hive-metastore-create-tables/hive-site.xml

in the File search for:

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:postgresql://myhost/metastore</value>
</property>

and change to the correct IP that is your Master Hadoop Server where u are running Cloudera Manager also every link in that file that is not correctly write to the hadoop master Cloudera manager connector , you will have to change to the correct IP

after all this just get back to the autoinstall of cloudera manager and run again and it will be all good :)

that it all the installation that you have to work around our contract cloudera support (that's their business) :)

all this it works fine for me when i have this problem in de cloudera CDH 4.X with sorl

Regards