Problems with installing Oracle DB 12c EE, ORA-12547: TNS: lost contact

For development purposes I wanted to install Oracle Database 12c Enterprise Edition to Vagrant box so that I could play with it. It should’ve gone quite straight forwardly but in my case things got complicated although I had Oracle Linux and the pre-requirements fulfilled. Everything went fine until it was time to run the DBCA and create the database.

The DBCA gave “ORA-12547: TNS: lost contact” error which is quite common. Google gave me couple of resources to debug the issue. Oracle DBA Blog explained common issues which cause ORA-12547 and solutions to fix it.

One of the suggested solutions was to check to ensure that the following two files are not 0 bytes:

ls -lt $ORACLE_HOME/bin/oracle
ls -lt $ORACLE_HOME/rdbms/lib/config.o

And true, my oracle binary was 0 bytes

-rwsr-s--x 1 oracle oinstall 0 Jul  7  2014 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

To fix the binary you need to relink it and to do that rename the following file:

$ cd $ORACLE_HOME/rdbms/lib
$ mv config.o config.o.bad

Then, shutdown the database and listener and then “relink all”

$ relink all

If just things were that easy. Unfortunately relinking ended on error:

[oracle@oradb12c lib]$ relink all
/u01/app/oracle/product/12.1.0/dbhome_1/bin/relink: line 168: 13794 Segmentation fault      $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/install/modmakedeps.pl $ORACLE_HOME $ORACLE_HOME/inventory/make/makeorder.xml > $CURR_MAKEORDER
writing relink log to: /u01/app/oracle/product/12.1.0/dbhome_1/install/relink.log

After googling some more I found similar problem and solution: Relink the executables by running make install.

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk install
 
cd $ORACLE_HOME/network/lib
make -f ins_net_server.mk install
</re>
 
If needed you can also relink other executables:
<pre lang="shell">
make -kf ins_sqlplus.mk install (in $ORACLE_HOME/sqlplus/lib)
make -kf ins_reports60w.mk install (on CCMgr server)
make -kf ins_forms60w.install (on Forms/Web server)

But of course it didn’t work out of the box and failed to error:

/bin/ld: cannot find -ljavavm12
collect2: error: ld returned 1 exit status
make: *** [/u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/oracle] Error 1

The solution is to copy the libjavavm12.a to under $ORACLE_HOME lib as explained:

cp $ORACLE_HOME/javavm/jdk/jdk6/lib/libjavavm12.a $ORACLE_HOME/lib/

Run the make install commands from above again and you should’ve working oracle binary:

-rwsr-s--x 1 oracle oinstall 323649826 Feb 17 16:27 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

After this I ran the relink again which worked and also the install of the database worked fine.

cd $ORACLE_HOME/bin
relink all

Start the listener:

lsnrctl start LISTENER

Create the database:

dbca -silent -responseFile $ORACLE_BASE/installation/dbca.rsp

The problems I encountered while installing Oracle Database 12c Enterprise Edition to Oracle Linux 7 although in Vagrant and with Ansible were surprising as you would think that on certified platform it should just work. If I would’ve been using CentOS or Ubuntu it would’ve been totally different issue.

You can see the Ansible tasks I did to get Oracle DB 12c EE installed on Oracle Linux 7 in my vagrant-experiments GitHub repo.

Oracle DB 12c EE Ansible Tasks
Oracle DB 12c EE Ansible Tasks

Using Let’s Encrypt SSL certificates on Centos 6

Let's Encrypt all the things

Let’s Encrypt is now in public beta, meaning, you can get valid, trusted SSL certificates for your domains for free. Free SSL certificates for everyone! As Let’s Encrypt is relatively easy to setup, there’s now no reason not to use HTTPS for your sites. The needed steps are described in the documentation and here’s short guide how to setup Let’s Encrypt in CentOS 6.x and automate the SSL certificate renewal.

Let’s Encrypt installation

The Let’s Encrypt Client is a fully-featured, extensible client for the Let’s Encrypt CA that can automate the tasks of obtaining certificates and configuring web servers to use them. The installation is simple but in my case on CentOS 6.x I first needed to update to Python 2.7 as Let’s Encrypt supports Python 2.7+ only.

Installing Python 2.7 in Centos 6.x

# Install Epel Repository
yum install epel-release
 
# Install IUS Repository
rpm -ivh https://rhel6.iuscommunity.org/ius-release.rpm
 
# Install Python 2.7 and Git
yum --enablerepo=ius install python27 python27-devel python27-pip python27-setuptools python27-virtualenv -y

Setting up Lets encrypt

Install Git if you don’t have it yet.

yum install git

If letsencrypt is packaged for your operating system, you can install it from there, and the other solution is to use the letsencrypt-auto wrapper script, which obtains some dependencies from your operating system and puts others in a python virtual environment:

# Get letsencrypt
git clone https://github.com/letsencrypt/letsencrypt
 
# See help
./letsencrypt/letsencrypt-auto --help

Running the client

You can either just run letsencrypt-auto or letsencrypt, and the client will guide you through the process of obtaining and installing certs interactively or you you can tell it exactly what you want it to do from the command line.

For example obtain a cert for your domain using the Apache plugin to both obtain and install the certs, you could do this:

./letsencrypt-auto --apache -d thing.com -d www.thing.com -d otherthing.net

(The first time you run the command, it will make an account, and ask for an email and agreement to the Let’s Encrypt Subscriber Agreement; you can automate those with –email and –agree-tos)

Although you can use the Apache plugin to obtain and install the certs it didn’t work for me. I got an error: “The apache plugin is not working; there may be problems with your existing configuration.” This seems to be an issue with Apache 2.2 and until it’s fixed you can use the webroot authentication method as explained in documentation.

./letsencrypt-auto certonly --webroot -w /var/www/example/ -d example.com

The webroot plugin works by creating a temporary file for each of your requested domains in ${webroot-path}/.well-known/acme-challenge. Then the Let’s Encrypt validation server makes HTTP requests to validate that the DNS for each requested domain resolves to the server running letsencrypt. Note that to use the webroot plugin, your server must be configured to serve files from hidden directories.

Now your certificate and chain have been saved at Let’s Encrypt configuration directory at “/etc/letsencrypt” and “/etc/letsencrypt/live/ contains symlinks to the latest certificates. Making regular backups of this folder is ideal.

All we have to do now is set it up in Apache.

Configure Apache to use Let’s Encrypt certs

In Let’s Encrypt configuration directory at “/etc/letsencrypt/live/ the .pem files are as follows (from the Letsencrypt documentation):

  • privkey.pem: Private key for the certificate.
    • This must be kept secret at all times! Never share it with anyone, including Let’s Encrypt developers. You cannot put it into a safe, however – your server still needs to access this file in order for SSL/TLS to work.
    • This is what Apache needs for SSLCertificateKeyFile
  • cert.pem: Server certificate only.
    • This is what Apache needs for SSLCertificateFile.
  • chain.pem: All certificates that need to be served by the browser excluding server certificate, i.e. root and intermediate certificates only.
    • This is what Apache needs for SSLCertificateChainFile.
  • fullchain.pem: All certificates, including server certificate. This is concatenation of chain.pem and cert.pem.

Now that we know which file is which we can configure our VirtualHost to use SSL with our new certs. Change the following lines in your Apache’s virtualhost’s SSL configuration:

...
SSLCertificateFile /etc/letsencrypt/live/<your-domain>/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/<your-domain>/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/<your-domain>/chain.pem
...

Finally, restart apache

You can test that your SSL is working with SSL Labs.

Automate updating Let’s Encrypt certs

As you surely noticed Let’s Encrypt CA issues short lived certificates (90 days) and you have to renew the certificates at least once in 3 months. Nice way to force sysadmins to automate the process.

To obtain a new version of the certificate you can simply run Let’s Encrypt again but doing that manually is not feasible. Let’s Encrypt is working hard on automating the renewal process but until that we have to do it by ourselves.

Fortunately we don’t need to invent our own scripts as there’s excellent article about automating Let’s Encrypt and script for crontab.

Get the autole.sh -script from GitHub that automates tasks like:

  • Check the expire date of the certificate and renew when the remaining days are below a value
  • Check that the directory for the challenge is well mapped
  • Alert the admin if it’s not possible to renew the certificate

Now you can renew certain domain’s certificates with

./autole.sh www.mydomain.com

And to renew all your certificates use

./autole.sh --renew-all

Now you can add this to the crontab, run weekly, and your certificates will be ready and renew automatically. This cron job will execute the command every Monday at 08:30.

30 8 * * 1 /usr/local/sbin/autole.sh <your-domain> >> /var/log/autole.log

Now before I switch my WordPress over to HTTPS I have to do some find & replace in the database and fix the URL’s of the images to be protocol relative.