Authentication with LDAP provider in WebLogic gets stuck

Lately we upgraded our Java EE applications to new platform and began seeing stuck threads and slow starting times. The platform was upgraded from OC4J to WebLogic 12c and also the underlying LDAP service was changed to Oracle Access Manager. Looking at the server logs the one possible reason for stuck threads was quite clear: LDAP requests.

Fortunately the stuck threads problem with LDAP was a known problem with Oracle Weblogic Server 10.3.2 and later and covered in Oracle Support doc 1436044.1. The LDAP provider fails to authenticate for some users and the server logs show Stuck Threads in LDAP requests:

<10.9.2014 11.43.46 EEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "607" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 607304 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
        java.lang.Object.wait(Native Method)
        java.lang.Object.wait(Object.java:503)
        netscape.ldap.LDAPMessageQueue.waitForMessage(Unknown Source)
        netscape.ldap.LDAPMessageQueue.waitFirstMessage(Unknown Source)
        netscape.ldap.LDAPConnection.sendRequest(Unknown Source)
        netscape.ldap.LDAPConnection.search(Unknown Source)
        weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3771)
        weblogic.security.providers.authentication.LDAPAtnDelegate.userExists(LDAPAtnDelegate.java:2384)
        weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:199)
        com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
        java.security.AccessController.doPrivileged(Native Method)
        com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:606)
        javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
        javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
        javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
        javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
        javax.security.auth.login.LoginContext.login(LoginContext.java:595)
        ...

The cause for this is that authentication requests are hanging whenever the LDAP server is slow. By default, connections and searches to the LDAP server do not time out, so if the LDAP server is slow, authentication requests may take a very long time to retry. This can be seen as many threads stuck doing LDAP searches.

The solution is to set a timeout on LDAP requests for example as below (described in Oracle Support doc 1436044.1):

  1. Log in to the WLS Administration Console.
  2. Navigate to Security Realms -> myrealm -> Providers -> “your_ldap_authenticator.”
  3. Select the following values:
    • Connect Timeout 30
    • Results Time Limit 5000
    • Uncheck “Keep Alive Enabled”
  4. Save and apply changes. Restart the required servers if prompted.

NOTE: The optimal values may differ from environment to environment. But we can try the values specified here as starting places, and they will help in most cases like this. Our original values for the LDAP authenticator settings were: Connect Timeout: 0; Results Time Limit: 0; Keep Alive Enabled unchecked.

This is still a partial solution as you should investigate why the LDAP is slow. For now this solves our problem but has some side effects with user authentication.

Planning for Fujitsu Forum 2014: Human Centric Innovation

In late November it’s again time for business- and IT decision-makers, experts, project managers, IT architects, consultants and bloggers to travel to Munich, Germany to visit Fujitsu’s largest IT-event in Europe: Fujitsu Forum 2014. This year the event is held 19th and 20th November at the ICM in Munich and there will probably be over 10,000 attendees around the world getting insights and looking for strategic and operative ways to modernize and develop their own IT or the IT landscapes of their customers. Last year I was one of the bloggers invited by Fujitsu to visit Fujitsu Forum 2013 and it was an insightful experience.

The motto of the Fujitsu Forum 2014 is “Human Centric Innovation which expresses focus on a safe and prosperous future through innovations in information and communication technology. The event description tells us that these ICT innovations strongly support enterprises, public organizations and individuals in creating wealth and value. In overall Fujitsu Forum is a good place to find innovations that reduce costs and risks as well as increase agility and improve efficiency. Big and important words but what does it mean in practice? The event overview with summary about keynotes and breakout sessions gives you some idea about the different topics covered so let’s take a short look at the events I would select to attend.

There are a lot of simultaneous events so to make it easier to organize the events of your choice there’s an excellent schedule builder. The presentations in in the list of Keynotes and Breakout sessions are divided in three conference tracks: “Human Empowerment”, “Connected Infrastructure”, “Creative Intelligence”. The topics cover themes like how to design the future, workplace of 2020, tablet meets notebook, digital transformation, wearable technologies, Internet of Things and connected vehicles. Overall I selected 14 sessions to my agenda but pruned it to three keynotes and five breakout sessions as you don’t have time for every interesting topic.

From Fujitsu Forum 2013: Keynote
From Fujitsu Forum 2013: Breakout session

Keynote: Human Centric Innovation & how to design the future by Tango Matsumoto and Brian Johnson (Intel), 19 November, 11:00, Auditorium

“How value can be generated for your business and for our society by the new Human Centric Innovation approach. Matsumoto will explain how Fujitsu contributes to business growth and the resolution of social challenges which will set the scene for all the subsequent presentations.”

“What kind of future do you want to live in? What are you excited about and what concerns you? What is your request of the future? Brian Johnson answers these questions and more with The Tomorrow Project, a fascinating initiative to investigate not only the future of computing but also the broader implications on our lives and planet.”

Workplace Anywhere – Increasing enterprise productivity by David Rosewell, Simon Gray and Thomas Zell, 19 November, 13:00, Room 13b

“Organizations are mobilizing the enterprise to deliver both increased productivity and cost savings. They seek to empower and enable their people to get the job done and look to embrace more flexible working practices to maximize productivity. We will guide you through the options and a vision for the 2020 workplace to identify your ideal workplace strategy.”

Keynote: Fujitsu and its customers by Jürgen Walter, 19 November, 14:00, Auditorium

“Human Centric Innovation is for a world where technology complements all of our lives. A place where information continually delivers knowledge and innovation thrives. How does Fujitsu implement this vision and how do customers benefit from it? Jürgen Walter will address these questions, depicting some remarkable customer examples.”

Fujitsu Laboratories Group’s R&D vision and key initiatives by Hideyuki Saso, 19 November, 15:00, Room 13b

“Fujitsu Laboratories’ R&D vision, highlighting key activities, under a mission to conduct R&D from a mid to long-term perspective, to generate new business models and discover new markets to drive Fujitsu’s business. We backcast future markets, products and services, engaging in trend-conscious R&D of: platform/applied/verification technologies, ubiquitous computing domain human interfaces and devices, data-leveraging domain knowledge platforms and cybersecurity, ICT platform domain computing and networking, software-supporting product development and manufacturing, and electronic devices. By interlinking such technologies, we aim to drive global business.”

Keynote: Digital Transformation & Fujitsu in Society by Joseph Reger and Duncan Tait, 20 November, 10:00, Auditorium

“As the Internet and other information and communication technologies penetrate all areas and aspects of life, business and infrastructure, a hyper-connected world is created. Digital and analog businesses, processes and, indeed, worlds converge on the basis of ICT technologies. Innovation accelerates, new value propositions and new businesses are created, existing businesses fundamentally transformed. What this development means, what technologies it requires and will create and how to not just cope with it but how to make good use of it, is the subject of the presentation.”

“Fujitsu uses the power of technology to contribute to the development of sustainable societies around the world. Working with both businesses and governments, in fields as diverse as energy, transportation, food, health, the environment, and education, Fujitsu’s ICT can drive social innovation and generate solutions. Duncan will present highlights from this area of Fujitsu’s activities and show that the company’s vision of a Human Centric Intelligent Society is already becoming a reality.”

The human-centric workplace: Joy and efficiency got married by Christian Bock, Markus Seifart and Jeffrey Shomper, 20 November, 12:00, Room 13b,

“A successful workplace IT strategy is all about people. Generation Y and BYOD drive a new understanding of workplace IT, one that puts the user at the center and embraces individuality. Progressive companies realize that this new thinking boosts both productivity and employee satisfaction. Fujitsu shares best practices that empower users to achieve their full potential.”

Wearable technologies for human empowerment by Naoyuki Sawazaki, 20 November, 13:00, Room 12

“Although various types of wearable devices such as eyeglasses, watches and other gadgets for health and fitness are gaining more attention in consumer markets, the true potential of wearable technologies lies in enterprise or business use. Because they enable users to get ICT support in a hands-free manner, they offer a clear advantage especially for workers in factory or building maintenance and other onsite operations to make their work more efficient, with fewer errors and oversights, even when they are not experts. In this session, current trends of wearable technologies are briefly reviewed, and then, research activities in Fujitsu Laboratories including the newly developed glove-style wearable device are introduced.”

Exploiting IoT & Hyperconnectivity – A Life & Death Example by Joseph Reger and Antonio Jara, 20 November, 15:00, Room 13b

“The Internet of Things promises to create huge new opportunities in business and society. In this session we demonstrate the use of Fujitsu RunMyProcess to connect wearables, mobiles, cloud software and physical sensors in order to show how time saved through hyperconnectivity could literally mean the difference between life and death for a critically injured cyclist.”

And of course there’s a comprehensive exhibition of products and solutions which reveals e.g. how data center and client landscapes can be improved through innovation initiatives. Though this year I’m not sure what to expect and look forward to regarding Fujitsu’s Ultrabooks and tablets. Anyways the exhibition offered by Fujitsu and its partners has always been interesting. Last year we got i.a. hands-on with U904 and T904 Ultrabooks and tested the 360 degree video conferences and collaboration setup.

From Fujitsu Forum 2013: Intelligent desk
From Fujitsu Forum 2013: 360 degree video conferences

We all love technical topics and talks but there’s time also for refreshments and entertainment. On Wednesday evening there’s an opportunity to enjoy Oktoberfest themed evening event. It’s a nice entertainment and networking opportunity with drinks and special Bavarian dishes. Last year there was crispy roast chicken (Hendl), spiced strips of pork belly (Hüttenspeck), bread and dripping (Schmalzbrot), chive bread, savory cheese spread with pretzels (Obatzda und Brezn). Hmmm, and now I’m again hungry :)

Fujitsu Forum with keynotes, breakout sessions and exhibition looks interesting and also this year I’m one of the Fujitsu Digital Influencer program’s bloggers invited to visit Fujitsu Forum 2014. It will be exciting to see the latest technology trends and other bloggers and Master your Business campaign team. In other news I’m again taking part in Fujitsu’s Master Your Business campaign which starts on November 6th. The campaign will be fun as I just got the device I’m testing and it’s pretty sweet.

Until then, read about my insights from last year’s Fujitsu Forum 2013 where I was invited by Fujitsu and follow me on Twitter (@walokra) for technology insights and maybe we will see on 18th and 20th November at the ICM in Munich.

Distribute project’s artifacts in Maven Central with OSSRH

You have developed some crafty Java library or Maven plugin and now you want to distribute it to other users through Maven Central repository? Using Sonatype Open Source Software Repository Hosting Service is a nice way for open source projects to achieve that and there’s two options to get your artifacts in it: 1. Release with Sonatype’s process or 2. Use Bintray’s process. In short, both options end up with the same results, your project’s artifacts are in Maven Central repository, but using Sonatype’s process and Maven release plugin makes it easy to release new version after you have set it up.

Using Sonatype’s service for pushing artifacts to Maven Central

Sonatype Open Source Software Repository Hosting Service (OSSRH) provides Maven repository hosting service for open source projects. You can deploy snapshots, stage releases, and promote your releases so they will be synced to Maven Central.

The process with Sonatype has a bit more steps than Bintray but if you follow the user guide, quite easy. And now as they have made some changes to their release process it’s easier more or less the same than using Bintray. All you need to do is to sign up a Sonatype JIRA account, create a JIRA ticket and make some POM/settings configuration and then use Maven release plugin to perform release. The procedure is described in Sonatype OSS Maven Repository Usage Guide but here is a short recap.

Performing release deployment with the Maven release plugin

Using Maven to perform release deployment to OSSRH is described in Sonatype’s guide which tells you how to do it manually or through the Maven release plugin which I find convenient.

First start with Initial Setup and create your JIRA account and new project ticket. The new project ticket triggers creation of your repositories. While your repositories are taken care of you should review the requirements for components in Central.

The artifacts need to be GPG signed so if you yet haven’t got a GPG key follow this guide to create one. Then you need to have your GPG key in a keyserver so start by sending your GPG key to keyserver:

gpg --keyserver pool.sks-keyservers.net --send-keys <your KEYID>

If you have more than one GPG key in your keychain then it’s useful to set a default key to be used for signing. Edit your ~/.gnupg/gpg.conf and uncomment the “default-key KEYID” line and add your KEYID.

Your Maven project settings

Configure your Maven settings.xml to have Sonatype OSSRH credentials (in .m2/settings.xml).

<settings>
     ...
    <servers>
    <server>
      <id>ossrh</id>
      <username>your-jira-id</username>
      <password>your-jira-pwd</password>
    </server>
  </servers>
     ...
</settings>

Edit your’project’s pom.xml for Maven release

Add the distributionManagement section to point to OSSRH.

<distributionManagement>
	<snapshotRepository>
		<id>ossrh</id>
		<url>https://oss.sonatype.org/content/repositories/snapshots</url>
	</snapshotRepository>
	<repository>
		<id>ossrh</id>
		<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
	</repository>
</distributionManagement>

The configuration for the Maven release plugin should include disabling the release profile that is part of the Maven Super POM, since we are using our own profile, and specify the deploy goal together with the activation of our release profile

<pluginManagement>
          <plugins>
                <plugin>
                     <groupId>org.apache.maven.plugins</groupId>
                     <artifactId>maven-release-plugin</artifactId>
                     <version>2.5</version>
                     <configuration>
                          <useReleaseProfile>false</useReleaseProfile>
                          <releaseProfiles>release</releaseProfiles>
                          <goals>deploy</goals>
                     </configuration>
                </plugin>
          </plugins>
</pluginManagement>

Add the nexus-staging plugin to define where’s Nexus and if we release it right after closing.

<plugin>
	<groupId>org.sonatype.plugins</groupId>
	<artifactId>nexus-staging-maven-plugin</artifactId>
	<version>1.6.3</version>
	<extensions>true</extensions>
	<configuration>
		<serverId>ossrh</serverId>
		<nexusUrl>https://oss.sonatype.org/</nexusUrl>
		<autoReleaseAfterClose>true</autoReleaseAfterClose>
	</configuration>
</plugin>

Add the release profile for signing with GPG & Javadoc and Sources Attachments

<profile>
	<id>release</id>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-source-plugin</artifactId>
				<version>2.2.1</version>
				<executions>
					<execution>
						<id>attach-sources</id>
						<goals>
							<goal>jar-no-fork</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-javadoc-plugin</artifactId>
				<version>2.9.1</version>
				<executions>
					<execution>
						<id>attach-javadocs</id>
						<goals>
							<goal>jar</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-gpg-plugin</artifactId>
				<version>1.5</version>
				<executions>
					<execution>
						<id>sign-artifacts</id>
						<phase>verify</phase>
						<goals>
							<goal>sign</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</profile>

You can see the full pom.xml e.g from my markdown-page-generator-plugin’s pom.xml.

Publishing Snapshots

Snapshot deployment are performed when your version ends in -SNAPSHOT. You do not need to fulfil the requirements when performing snapshot deployments and can simply run

$ mvn clean deploy

Successfully deployed SNAPSHOT versions will be found in Snapshot repository. If you need to delete your artifacts, you can log in to Sonatype’s Nexus using the same credentials you use to access to the Sonatype JIRA.

Performing a Release Deployment

The Maven Release Plugin can be used to automate the changes to the Maven POM files, sanity checks, the SCM operations required, the actual deployment execution and you can perform a release deployment to OSSRH with the following steps.

Prepare the release by answering the prompts for versions and tags

$ mvn release:clean release:prepare

If prepare fails use

$ mvn release:rollback
$ mvn release:clean

The prepare will create a new tag in SCM, even in GitHub, automatically checking in on your behalf. For it to work you need to have working public key in GitHub for git-push.

Perform the release. The perform process will ask for your gpg.passphrase if you don’t give it with passphrase argument.

$ mvn release:perform

For some reason I can’t provide the GPG passphrase when Maven GPG plugin asks it and I have to use the -Dgpg.passphrase argument.

$ mvn -Darguments="-Dgpg.passphrase=&lt;your GPG key passphrase&gt; release:perform

If release:perform fails to error “gpg: signing failed: No pinentry” then on macOS you need to restart gpg-agent and define pinentry-program. gnupg21 comes with pineentry as dependency. Or see Gist about commit signing with GPG on macOS.

$ killall gpg-agent && gpg-agent --daemon --use-standard-socket --pinentry-program /usr/local/bin/pinentry

This execution will deploy to OSSRH and release to the Central Repository in one go, thanks to the usage of the Nexus Staging Maven Plugin with autoReleaseAfterClose set to true.

Now your artifacts are in the Releases repository. The updates to The Central Repository search can take up to two hours. Once your artifacts are released you will not be allowed to update or delete them.

The first time you promote a release, you need to comment on the OSSRH JIRA ticket you created so OSSRH can know you are ready to be synced.

Manually Releasing the Deployment to the Central Repository

If you don’t want to release artifacts automatically you can also do it manually. Set the autoReleaseAfterClose set to false so you can inspect and potentially release the deployed artifacts manually. Then to release your artifacts is done through Nexus as described in the guide.

Using Bintray for pushing artifacts to Maven Central

Bintray offers developers the fastest way to publish and consume OSS software releases. Whether you are distributing software packages or downloading ones”. In short it provides i.a. an alternative way to release artifacts to Maven Central and to Sonatype Open Source Software Repository.

I found out about Bintray after I had already used the Sonatype’s way to release artifacts so I haven’t tested this myself. Bintray’s process is more or less similar than Sonatype’s but if I hadn’t read the blog post about how to do it I wouldn’t have know where to start. In short the process is the following.

  1. Register to Bintray and set up auto-signing: Generate yourself a keypair, if you don’t have one. Add it to your profile, and setup your default Maven repo (or a new one) for signing with your GPG key: Bintray can then sign your jars automatically.
  2. Add your Sonatype account under “accounts”. If you don’t have one, follow this procedure
  3. Create and link your package: Import from a GitHub repo or create a new package for your Maven project.
  4. You can link your package to JCenter by clicking “Add to JCenter”.
  5. Set up Maven up to deploy to Bintray by copy-pasting the pom.xml snippets from “Set me up!” guide.

Then for each release:

  1. Deploy: Deploy files by running your build tool.
  2. Publish: Review the build artifacts and publish the version files if satisfied.
  3. Sync: On the version page go to the Maven Central tab, enter your Sonatype password and click “Sync” and you’re done!
  4. Your package is now in https://oss.sonatype.org/content/repositories/releases and will be synced to Maven Central (usually takes time). In case of a sync problem, Bintray will automatically take care of any needed cleanup.

Summary

For distributing your open source projects’ artifacts through Maven Central repository and OSSRH you have two options: 1. Release with Sonatype’s process or 2. Use Bintray’s process. Both options end up with the same results: your project’s artifacts are in Maven Central repository. In my opinion if you’re using Maven then it doesn’t really matter which option you choose.

Before Sonatype simplified their process for making releases to OSSRH the Bintray’s process was the only almost pain-free gateway to Maven Central. But with the new Sonatype way the two alternative ways are more or less the same and I could say that Sonatype’s new process feels a bit easier when using the Maven release plugin. In practice there’s not much differences, e.g. both supports SNAPSHOTs, Sonatype in OSSRH and Bintray in oss.jfrog.org. Bintray is more flexible especially if not using Maven so if you’re using other tools like Gradle then it’s the easier way to go.