Using Apache for proxying connections to Crucible

Atlassian’s web applications are great tools for software development and they are relatively easy to setup because they come with Jetty servlet container and HQSQL database. You only have to install Java. Some of the applications can be also run like any normal deployable WAR-packaged web application for example with Apache Tomcat which gives you more control and administration options. But unfortunately code review tool Crucible isn’t one of those applications and maybe will never be.

Proxying connections to Crucible

By default Crucible runs in port 8060 which isn’t nicely looking for users. It’s way better to use ports 80 or 443 which are normal HTTP and HTTPS ports and are omitted from browser’s address bar. Of course you can configure that in the Administration screens, or by editing Crucible’s config.xml and restarting Crucible but if you run Crucible as a non root or also have other software running on the same server that isn’t an option.

One solution is to use Apache HTTP server to proxy connections from port 443 to Crucible’s listening port. I did it for Crucible and FishEye on CentOS x86_64 but things are mostly the same also on other Linuxes. I also disabled the HTTP port and used just the SSL enabled HTTPS with self generated certificates.

First we setup Apache for proxying connections to Crucible and then we generate some SSL certificates for HTTPS. If you haven’t Apache installed you can do it with yum like: yum install httpd.x86_64 mod_ssl openssl

1. Set HTTPS proxying in /etc/httpd/conf.d/ssl.conf

...

SSLProxyEngine on
ProxyRequests Off
ProxyPreserveHost On
ProxyPassReverse /crucible ajp://127.0.0.1:8060/crucible
proxyPass /crucible ajp://127.0.0.1:8060/crucible

RewriteEngine On
...

2. Generating SSL Certificate for Apache

# openssl genrsa -out localhost.key 1024
# openssl req -new -key localhost.key -out localhost.csr
# openssl x509 -req -days 365 -in localhost.csr -signkey localhost.key -out localhost.crt
# mv localhost.csr localhost.key /etc/pki/tls/private/
# mv localhost.crt /etc/pki/tls/certs/

3. Start httpd

# service httpd start

Configuring Crucible

4. Configure Crucible (http://hostname:8060/admin)

Edit Web Settings:
-----------------
Web context: crucible
Http Bind: (none)
Ajp13 Bind Address: ajp://127.0.0.1:8060/crucible

And you’re ready.

Running FishEye & Crucible as a service in Linux

Atlassian’s tools for supporting software development are great but they aren’t really admin friendly to start with. For example FishEye & Crucible doesn’t ship with scripts to start it at system boot time but with the help of Atlassian’s Wiki, sysadmin tasks and scripts you can run it as a normal service. First we create a dedicated user for crucible and second we add a new service for it. I have done this on CentOS 5.7 x86_64.

Setting up the service account

As the root user, create a separate “FishEye & Crucible” service account at root shell:

# useradd -c "FishEye & Crucible service account" -d /home/crucible -m crucible

To make it easier for this to work also after FishEye & Crucible upgrades we create a symbolic link to the latest version (modify “/opt/fecru” to match your deployment).

# ln -s /opt/fecru/fecru-2.7.15 /opt/fecru/latest

Then, ensure that this user is the filesystem owner of the FishEye & Crucible instance (modify “/opt/fecru” to match your deployment).

# chown -R crucible:crucible /opt/fecru

Running Crucible as a crucible user

Save the following script to /etc/init.d/crucible. Be sure to edit the FISHEYE_HOME value to the location where your FishEye/Crucible instance resides:

#!/bin/bash
# RUN_AS: The user to run fisheye & crucible as. Its recommended that you create a separate user account for security reasons
RUN_AS=crucible
 
# FISHEYE_HOME: The path to the FishEye & Crucible installation. Its recommended to create a symbolic link to the latest version so the process will still work after upgrades.
FISHEYE_HOME="/opt/fecru/latest"
# FISHEYE_INST: The path where the data itself will be stored.
export FISHEYE_INST="/opt/fecru/fecru-data"

fisheyectl() {
        if [ "x$USER" != "x$RUN_AS" ]; then
                # If running without FISHEYE_INST
                # su - "$RUN_AS" -c "$FISHEYE_HOME/bin/fisheyectl.sh $1"
                su - "$RUN_AS" -c "FISHEYE_INST=$FISHEYE_INST $FISHEYE_HOME/bin/fisheyectl.sh $1"
        else
                "$FISHEYE_HOME/bin/fisheyectl.sh $1"
        fi
} 

case "$1" in
        start)
                fisheyectl start
                ;;
        stop)
                fisheyectl stop
                ;;
        restart)
                fisheyectl stop
                sleep 10
                fisheyectl start
                ;;
        *)
                echo "Usage: $0 {start|stop|restart}"
esac
 
exit 0

After saving the script, modify it’s permissions so that it can be executed:

# chmod 755 /etc/init.d/crucible

Running Crucible as a service

Now that we have an init script we can add it as a service and be able to configure the system to run the script on startup (more precisely, ensure that Crucible runs in runlevels 3, 4 and 5):

chkconfig --add crucible
chkconfig crucible on

Verify that the script has been installed correctly:

# chkconfig --list crucible

After this has been done you can manually start or stop the service by using these commands:

service crucible stop
service crucible start

And you’re done.

Using CAcert.org signed certificates for TLS

Setting up Transport Layer Security (TLS), or as previously known as Secure Sockets Layer (SSL), for Apache, Postfix and IMAP like Dovecot is fairly easy. You just need some digital certificates and configuration. If you don’t want to pay for certificates from trusted sources like Thawte or you just don’t need that kind of trust (for development purposes), you can always produce your own certificates. But there is also a middle way: using CAcert.org signed certificates.

Background
Wikipedia tells us that CAcert.org is a community-driven certificate authority that issues free public key certificates. CAcert automatically signs certificates for email addresses controlled by the requester and for domains for which certain addresses (such as “hostmaster@example.com”) are controlled by the requester. Thus it operates as a robot certificate authority. CAcert certificates can be used like any other SSL certificates although they are considered weak because CAcert does not emit any information in the certificates other than the domain name or email address. To create higher-trust certificates, users can participate in a web of trust system whereby users physically meet and verify each other’s identities. They are also not as useful in web browsers as certificates issued by commercial CAs such as VeriSign, because most installed web browsers do not distribute CAcert’s root certificate. Thus, for most web users, a certificate signed by CAcert behaves like a self-signed certificate.

Generating Certificates
The procedure to sign your certificate at CAcert is rather simple. This guide assumes that the certificates are in /etc/ssl/cacert/ and you are as root.

0. Join CAcert.org and fill in your details. After email verification and login, add domain and service will try to verify that you can read mail on one of following accounts: root, hostmaster, postmaster, admin, webmaster or email addresses that can be found on whois data of domain that you provided.

1. Generate a private key that is not file encrypted:

openssl genrsa -out domainname.key 1024
chown root:root domainname.key
chmod 0400 domainname.key

Private keys should belong to “root” and be readable only by root.

You could also create a private key that is encrypted: openssl genrsa -des3 -out domainname.key 1024

2. Create a CSR with the RSA private key (output will be PEM format). Do not enter extra attributes at the prompt and leave the challenge password blank (press enter):

openssl req -new -key domainname.key -out domainname.csr

3. Verify the contents of the CSR or private key:

openssl req -noout -text -in domainname.csr
openssl rsa -noout -text -in domainname.key

4. Send your public key to be signed by and request new server certificate from CAcert.org web site (Class 1 certificate). When you are asked for CSR paste content of domainname.csr. It should look like this:

-----BEGIN CERTIFICATE REQUEST-----
MIIB3TCCAUYCAQAwgZwxCzAJBgNVBAYTAkZJMRAwDgYDVQQIEwdVdXNpbWFhMQ8w
...clip...
MQ==
-----END CERTIFICATE REQUEST-----

You can verify the content of request before sending it

openssl req -in domainname.csr -text -verify -noout

5. Copy the Server Certificate from the CAcert.org webpage and put it in domainname.crt file and add permissions.

chmod a=r domainname.crt

Check at least the contents of Validity and Subject fields:

openssl x509 -in domainname.crt -text -noout

6. Get CAcert.org root certificate

wget -nv https://www.cacert.org/certs/root.crt -O cacert-org.crt
chmod a=r cacert-org.crt

Check the contents:

openssl x509 -in cacert-org.crt -text -noout

After that you’re ready to configure your services like Apache, Postfix and Dovecot to use the new certificate. Read about it later.

Installing Apache Tomcat 6 on CentOS

CentOS is great substitute for Red Hat Enterprise Linux but is missing some useful packages like Apache Tomcat 6. Installing Apache Tomcat 6 on CentOS 5 from gzip-package is fairly easy. The following guide is at least for CentOS 5.4.

Pre-Requirements
First you need to install Sun JDK and you can follow the instructions given in Installing Sun JDK 1.6 on CentOS

After Java is on place it’s time to get ready for Tomcat.

Download Apache Ant and Tomcat

  1. Download apache-ant and apache-tomcat -packages.
  2. Extract those packages to /opt/
    • #[root@srv ~]# cd /opt
      # tar -xzf apache-tomcat-6.0.26.tar.gz
      # tar -xzf apache-ant-1.7.1-bin.tar.gz
      
  3. Create a symbolic link for Ant
    • # ln -s /opt/apache-ant-1.7.1/bin/ant /usr/bin/
      

Create start script

  1. Create a tomcat user so that we don’t need root privileges for Tomcat
    • # useradd -d /opt/apache-tomcat-6.0.26/ tomcat
      
  2. Create start script to /etc/init.d for starting and stopping Tomcat
    • #  vim /etc/init.d/tomcat
      
  3. The script is (via Build a safe cage for Tomcat)
    • 1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      
      #!/bin/bash
      #
      # tomcat       Starts Tomcat Java server.
      #
      #
      # chkconfig: 345 88 12
      # description: Tomcat is the server for 
      # Java servlet applications.
      ### BEGIN INIT INFO
      # Provides: $tomcat
      ### END INIT INFO
       
      JAVA_HOME=/usr/java/jdk1.6.0_18
      export JAVA_HOME
      TOMCAT_HOME=/opt/apache-tomcat-6.0.26/bin
      START_TOMCAT=/opt/apache-tomcat-6.0.26/bin/startup.sh
      STOP_TOMCAT=/opt/apache-tomcat-6.0.26/bin/shutdown.sh
       
      # Source function library.
      . /etc/init.d/functions
       
      [ -f $START_TOMCAT ] || exit 0
      [ -f $STOP_TOMCAT ] || exit 0
       
      RETVAL=0
       
      umask 077
       
      start() {
              echo -n $"Starting Tomcat Java server: "
              daemon su -c $START_TOMCAT tomcat
              echo
              return $RETVAL
      }
      stop() {
              echo -n $"Shutting down Tomcat Java server: "
              daemon su -c $STOP_TOMCAT tomcat
              echo
              return $RETVAL
      }
      restart() {
              stop
              start
      }
      case "$1" in
        start)
              start
              ;;
        stop)
              stop
              ;;
        restart|reload)
              restart
              ;;
        *)
              echo $"Usage: $0 {start|stop|restart}"
              exit 1
      esac
       
      exit $?
  4. Give executable rights for that script
    • # chmod 755 /etc/init.d/tomcat
      
  5. Add the script to CentOS services
    • # chkconfig --add tomcat
      
  6. Check the changes
    • # chkconfig --level 234 tomcat on
      # chkconfig --list tomcat
      
      tomcat 0:off 1:off 2:on 3:on 4:on 5:off 6:off
      
  7. You should see that the service uses levels 2, 3 and 4
  8. Test that the script is working and it gives no errors
    • # service tomcat start
      # service tomcat stop
      
  9. Everythings ready

Installing Sun JDK 1.6 on CentOS

CentOS doesn’t have a package for Sun JDK so it has to installed manually. It’s fairly easy but there are some steps to do that. This guide has been tested on CentOS 5.4 x64_86.

Step 1. Initial setup for building RPM
-!- Do this with a non-root user

  1. Create ~/.rpmmacros
    • $ vim ~/.rpmmacros
      %_topdir /home//rpmbuild
      %_tmppath %{_topdir}/tmp
      
  2. Create needed folders:
    • $ mkdir -p ~/rpmbuild/{SOURCES,SRPMS,SPECS,RPMS,tmp,BUILD}
      
  3. Build environment needs to be complete. Some needed packages are:
    • $ sudo yum install -y rpm-build gcc gcc-c++ redhat-rpm-config
      

Step 2. Installing your favorite JDK

  1. Download Sun JDK 1.6 update 14 from Sun Java download or the Sun JDK archive.
    • Choose the correct platform (for me it’s Linux x64) and download jdk-6u18-linux-x64-rpm.bin
  2. Give it executable rights: $ chmod 755 jdk-6u18-linux-x64-rpm.bin
  3. Run the binary to extract it into RPM form: $ ./jdk-6u18-linux-x64-rpm.bin
  4. Install it:
    • $ sudo rpm -Uvh jdk-6u18-linux-amd64.rpm
      
  5. Log out and in again to make the changes in the paths take effect
  6. Check the install
    • $ java -version
      java version "1.6.0_18"
      Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
      Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)
      
  7. Java is now installed on /usr/bin/java

Redirect HTTP and HTTPS traffic to Tomcat’s ports

Apache Tomcat likes with default settings to listen to requests on 8080 and 8443 ports but it is more enjoyable to use the more common 80 and 443 ports for HTTP and HTTPS traffic. This way the user don’t have to put those pesky port numbers after the address. Of course you could just tell Tomcat to listen to those ports but it has some negative sides: hassle with the startup and running Tomcat as root.

Luckily it is easy to tell the system to redirect the traffic from some port to other. Just define some new xinetd services in /etc/xinetd.d/tomcat.

# vim /etc/xinetd.d/tomcat
 
# Redirects any requests on port 80 to port 8080 (where Tomcat is listening)
service tomcat-http
{
        disable                 = no
        flags                   = REUSE
        wait                    = no
        user                    = root
        socket_type         = stream
        protocol                = tcp
        port                    = 80
        redirect                = localhost 8080
        log_on_success  -= PID HOST DURATION EXIT
 
        #per_source = UNLIMITED
        #instances = UNLIMITED
}
 
# Redirects any requests on port 443 to port 8443 (where Tomcat is listening)
service tomcat-https
{
        disable                 = no
        flags                   = REUSE
        wait                    = no
        user                    = root
        socket_type         = stream
        protocol                = tcp
        port                    = 443
        redirect                = localhost 8443
        log_on_success  -= PID HOST DURATION EXIT
 
        #per_source = UNLIMITED
        #instances = UNLIMITED
}

(via Securing Linux for Java services: The port dilemma)

Xinetd puts a connection limit per source IP, by default and this causes the service to become unresponsive when there are dozens of queries a second. You see the following kind of line in your messages log file: “xinetd[2049]: FAIL: tomcat-https per_source_limit from=123.456.789.123”. To correct this, uncomment the per_source and instances lines in your xinet.d file and restart it.

Also add those xinetd services to /etc/services.

# vim /etc/services
http        80/tcp     www www-http tomcat-http # WorldWideWeb http
http        80/udp     www www-http tomcat-http # WorldWideWeb HTTP
http        443/tcp    tomcat-https # WorldWideWeb HTTPS
http        443/udp    tomcat-https # WorldWideWeb HTTPS

And now just restart the xinetd and admire how your traffic is redirected to Tomcat’s ports.

# service xinetd restart

Force everything to transmit through HTTPS
If you also want to redirect all HTTP traffic to HTTPS you can add the following section to you Tomcat web.xml:

<web-resource-collection>
    <web-resource-name>Protected Context</web-resource-name>
    <url-pattern>/*</url-pattern>
</web-resource-collection>
<!-- auth-constraint goes here if you requre authentication -->
<user-data-constraint>
    <transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>

If you are using this redirection of all traffic to HTTPS with JIRA and want to attachments working also with Internet Explorer then you must add the following to your jira.xml (f. ex. /opt/tomcat/conf/Catalina/localhost/jira.xml). This is a Internet Explorer bug, for more information see http://jira.atlassian.com/browse/JRA-8179.

<Context ...>
...
<!-- for IE bug, see http://jira.atlassian.com/browse/JRA-8179-->
<Valve className="org.apache.catalina.authenticator.NonLoginAuthenticator"
disableProxyCaching="false" />
...
</Context>

Keeping up with the time in Xen

A simple tip for keeping up with the time in Xen when your domU isn’t syncing the clock frequently enough and your clock is whatever. Of course you can sync your clock with ntpdate but first you need to set the domain (domU) to run its wallclock independently from Xen.

Simply say in the command line of domU:
echo 1 > /proc/sys/xen/independent_wallclock

Or if you have noclobber on:
echo 1 >! /proc/sys/xen/independent_wallclock

To reenable tracking of Xen wallclock:
echo 0 > /proc/sys/xen/independent_wallclock

And to keep the setting between reboots, just add it to the /etc/sysctl.conf:
xen.independent_wallclock = 1

Although, it is better if the Xen dom0 syncs the clock frequently so all the domains (domU) don’t need to do it by themselves and thus wasting resources. Why do things multiple times when it could be done just once.

Samba performance problem after kernel update

Samba is a nice service to provide storage space through networks and it is relatively easy to set up. It’s not as fast as using NFS between Linux hosts but sometimes you don’t have that possibility. Sometimes there also might be quite confusing problems like I happened to notice.

My Samba service had worked fine for a long time but after a kernel update and a reboot, the performance was horrible. It was really slow to do anything. Fortunately I wasn’t the only one to come by with this problem and there was a topic Samba Performance Problem Due to Changing Linux Kernel in Samba Performance Tuning -guide which provided a solution to my problem. Just restarted the network interface and the performance was as good as before.

Earlier this month Samba Team Received Microsoft Protocol Docs so maybe in the near future we will get better and faster software.

The Protocol Freedom Information Foundation (PFIF), signed an agreement with Microsoft to receive the protocol documentation needed to fully interoperate with the Microsoft Windows workgroup server products and to make them available to Free Software projects such as Samba.

Microsoft was required to make this information available to competitors as part of the European Commission March 24th 2004 Decision in the antitrust lawsuit, after losing their appeal against that decision on September 17th 2007.
Samba Team