One of my client asked me how to monitor expiry dates of JKS certificate entries in a WebLogic domain. If an imported certificate entry expires in a JKS (Java Keystore File) then the HTTPS transactions is going to fail because of SSL handshake exception/failure.

So, a proactive administrator should monitor these certificates and should warn the 3rd party end point URL’s administrator or update/import the certificate with the new one before the failure. Especially, if you are working with OSB (Oracle Service Bus) and SOA Suite 11g/12c WebLogic domains it is very crucial to monitor imported JKS certificate entries.

 

 

Monitoring Certificate Expiry Dates of a JAVA Keystore (.JKS) File in a WebLogic Domain (Shell Script, Scheduling, Alerting)

How to monitor imported certificates (CRT, CER) expiry / expiration date in a JKS (Java Keystore) file and create remaining day alerts/notifications?

There are too many alternatives to do that. I have found some other scripts available but all of them based on JKS alias entries. Then decided to write a new one from scratch to iterate all JKS aliases in the JKS file.

I’ll explain two method for doing this JKS certificate monitoring in WebLogic.

 

• Shell scripting and OS Cron Job
• Shell script and WLSDM Scheduler

 

At the end of post I’ll put download link of the script and you can download it. Go to my GitHub project page to download entire project.

Step by Step Script Usage and Integration via Server CRON Job:

1. Download monitorJKS folder and upload to your server (Do it at the end, firstly read the whole tutorial)

2. Edit environment variables according to your server and WebLogic installation paths.

#!/bin/bash
#Author: M.Fevzi Korkutata | Last day of 2017 (New year PARTY script)
#Change environment variables:
__scriptPath=/home/oracle/scripts/monitorJKS
__keytool="/u01/weblogic/jdk8/jdk1.8.0_112/bin/keytool"
__keystore="/u01/weblogic/Middleware12212/wlserver/server/lib/DemoTrust.jks"
__keystorepass="DemoTrustKeyStorePassPhrase"
__thresholdDay="7"
__mailTo="[email protected] [email protected]"

 

3. Variable Info:

3.1.  Change “__thresholdDay” value according to your preference. By default, I have set it with 7 which means it will start to send ALERT emails 7 days before the certificate expiration.

3.2. Do not forget about to set your local  “__scriptPath” value.

4. Comment out below section for the SMTP email delivery. (Line 58)

#Comment out if you want to send as WARNING email.
#cat $__scriptPath/certificateExpireWarning.txt $__scriptPath/certificateSummary.txt | mail -s "!!! [WARNING] Check expired certificates !!!" $__mailTo

5. Create your CRON job for /path/to/monitorJKS/monitorJKSforWebLogic_Lnx.sh script. Running once in a day is just enough. Schedule for every midnight.

 

Step by Step Script Usage and Integration via WLSDM Scheduler:

Scheduling this script via WLSDM for WebLogic is more exciting and smarter. Because, I am going to use stdErr functionality of Shell and WLSDM interface.

1. Copy monitorJKSforWebLogic_Lnx.sh to server.

2. Go to “WLSDM Console > Configuration > Monitoring & Diagnostics > User Defined Actions / Scripts (tab)”  page.

3. In the page operations menu (on top-right of every page) click on “New User Defined Action / Script” button.

4. In the modal window add shell script file as below and save.

5. Switch to “Scheduled Jobs / Downtime” tab.

6. In the page operation menu (on top-right of every page) click on “New Scheduled Jobs / Downtime” button.

7. In the modal window add new CRON job as below and save. (you can choose every minute for testing the cron job temporarily)

 

That’s it! The integration and monitoring of JKS certificates expiry date is done. Let’s test it and see the results.

I chose every minute to test the script and understand that WLSDM scheduler is working properly. The threshold in the script is  100 days which is a very high value; in a production system it must be something like 5 or 7 days. In my test case the threshold is:

__thresholdDay=”100″ 

Let’s see the WLSDM scheduler output (enlarge below screen capture). Here is the job records and its results. It demonstrates all the JKS monitoring processes passed successfully and we will not get any email alert from WLSDM.

Drill down each monitoring cycle detail by clicking “View Output/Details” lens in the “Scheduled Job Execution History” list. WLSDM reflects all the “SHELL/BASH stdOut” and “echo/printf” message values in this modal window page. It is possible to see Linux/Unix/Windows shell/bash (terminal) scripts’ stdOut and stdErr values in this page. Click below screen capture to see a successful JKS monitoring output for my shell script/action which is monitorJKS.sh.

As you see the remaining days for the alias volthreadcom is 124 days. The threshold is 100 which means remaining days are more than threshold; so the result is as expected: –> Script executed successfully! Certificates are OK.

For the second scenario; I am going to change the threshold from 100 to 150. In this case volthreadcom JKS certificate entry would be in trouble 🙂 I am expecting to get a failed job result. Because the shell script will terminate itself with error (check “exit 1” in the script)

After updating threshold to 150; the WLSDM cron job switched to failed status immediately. Check below screen captures!

Click “View Details/Output” button to see what’s going on for the JKS certificate entries.

YESSS! Here is the most important part. I have received an email for the certificates that’s going to expire according to my threshold.

Click below image to enlarge and check the details for understanding how to receive HTML emails for the certificates going to expire.

 

Actually I am cheating WLSDM. Because; I know that WLSDM informs administrators about failed cron jobs/scripts by sending HTML email notifications as above. So, when I get an email about current “WLSDM WebLogic scheduled job” probably it means there is a WARNING about JKS certificate expiry date. Check below section in the script especially the (exit 1) part then you will understand what I mean. Otherwise please let me know to think about your case together 🙂

#__lcCEW = line count for certificate expire warning (file)
if [ $__lcCEW -gt 0 ]; then
(>&2 echo "!!! [WARNING] Check expired certificates !!!") 
(>&2 echo "$(cat $__scriptPath/certificateExpireWarning.txt)") 
#Comment out if you wanto send as WARNING email.
#cat $__scriptPath/certificateExpireWarning.txt $__scriptPath/certificateSummary.txt | mail -s "!!! [WARNING] Check expired certificates !!!" $__mailTo
exit 1

I have finished the documentation part and reading part. I have also prepared a Youtube tutorial/screencast about monitoring JKS entries. I’ve recorded the entire video on my iMac. The date function/command is different in MacOS (Sierra) than Linux. I have did some tricks for executing my shell script properly for macOS environment (If you are interested knock my door).

If you already read this technical blog post; please also watch the entire technical screencast it will inspire you about your PRODUCTION environment monitoring cases. Especially about WebLogic cron job integrations and definitions.

Youtube Screencast Tutorial: Monitoring Certificate Expiry Date of JKS

(Please share the screencast with other Oracle FMW WebLogic Community. Regards!) –> Sharing is just caring Fevzi Korkutata 🙂

Source Code (Script and sample output file):

Github URL: https://github.com/admineer/monitorJKSforWebLogic

All the project source and sample output files are available in the Github monitorJKSforWebLogic project. I am also adding the entire script as below. (Copy/Download the script from Github; because the shell script source file is well formated at there)

#!/bin/bash
#Author: M.Fevzi Korkutata | Last day of 2017 (New year PARTY script)

#Change environment variables: 
__scriptPath=/home/oracle/scripts/monitorJKS
__keytool="/u01/weblogic/jdk8/jdk1.8.0_112/bin/keytool"
__keystore="/u01/weblogic/Middleware12212/wlserver/server/lib/DemoTrust.jks"
__keystorepass="DemoTrustKeyStorePassPhrase"
__thresholdDay="100"
__mailTo="[email protected] [email protected]"

#Static Variables
__currentDate=$(date +%s)
__threshold=$(($__currentDate + ($__thresholdDay*24*60*60)))

#Flush output values
echo -n > $__scriptPath/certificateStatus.txt
echo -n > $__scriptPath/certificateExpireWarning.txt
echo -n > $__scriptPath/certificateSummary.txt

#Fetch certificate "until" dates
for i in $($__keytool -list -v -keystore $__keystore -storepass $__keystorepass | grep 'Alias name:' | perl -ne 'if(/name: (.*?)\n/) { print "$1\n"; }')
do
echo "$i valid until: "$($__keytool -list -v -keystore $__keystore -storepass $__keystorepass -alias "$i" | grep 'Valid from' | head -1 | perl -ne 'if(/until: (.*?)\n/) { print "$1\n"; }') >> $__scriptPath/certificateStatus.txt
done

#Calculate certificate remaining days 
__lc=$(cat $__scriptPath/certificateStatus.txt | wc -l)
for (( c=1 ; c<=$__lc ; c++ ))
do 
__alias=$(awk "NR==$c" $__scriptPath/certificateStatus.txt | awk '{print $1}')
__until=$(awk "NR==$c" $__scriptPath/certificateStatus.txt | perl -ne 'if(/until: (.*?)\n/) { print "$1\n"; }')
#echo $__until

__untilSeconds=`date -d "$__until" +%s`
__remainingDays=$(( ($__untilSeconds - $(date +%s)) / 60 / 60 / 24 ))

if [ $__threshold -le $__untilSeconds ]; then
#printf "[OK] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n"
printf "[OK] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" >> $__scriptPath/certificateSummary.txta
elif [ $__remainingDays -le 0 ]; then
#printf "[CRITICAL] ===> $__alias <=== !!! Certificate '$__alias' has already expired !!!\n"
printf "[CRITICAL] ===> $__alias <=== !!! Certificate '$__alias' has already expired !!!\n" >> $__scriptPath/certificateSummary.txt

else
#printf "[WARNING] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n"
printf "[WARNING] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" >> $__scriptPath/certificateSummary.txt
printf "[WARNING] ===> $__alias <=== Certificate '$__alias' expires in '$__until'! *** $__remainingDays day(s) remaining ***\n\n" >> $__scriptPath/certificateExpireWarning.txt
fi
done

#Decide on ALERT
__lcCEW=$(cat $__scriptPath/certificateExpireWarning.txt | wc -l)
if [ $__lcCEW -gt 0 ]; then
(>&2 echo "!!! [WARNING] Check expired certificates !!!") 
(>&2 echo "$(cat $__scriptPath/certificateExpireWarning.txt)") 
#Comment out if you want to send as WARNING email.
#cat $__scriptPath/certificateExpireWarning.txt $__scriptPath/certificateSummary.txt | mail -s "!!! [WARNING] Check expired certificates !!!" $__mailTo
exit 1 
else
echo "Script executed successfully! Certificates are OK!"
echo " "
echo "##################################################"
cat $__scriptPath/certificateSummary.txt
exit 0
fi