I had a customer contact me recently asking about identifying why one of their IBM i LPARs had so much additional disk space consumed compared to another LPAR that hosted the Domino cluster mates.  The primary server hosted 4 Domino servers and the secondary LPAR hosted only 2 Domino servers.  The two additional servers on the primary LPAR were both Sametime Community servers, which typically don’t consume much disk space.

I used the disk usage utility to collect information on how much disk space each Domino server consumed.  The output is in the table below.

The Domino servers on the primary LPAR were consuming 843.34 GB more disk space than the servers on the secondary LPAR.  The two Sametime servers were a minuscule portion of this  discrepancy.  There were some databases on the primary servers (MAIL03 and COLLAB03) that were not on the cluster mates, however those databases only accounted for about 150 GB of space.  So what was causing such a disparity in the amount of space the primary servers were consuming compared to the secondary servers?

The culprit turned out to be the data in the IBM_TECHNICAL_SUPPORT subdirectory of the MAIL03 server.  This subdirectory alone was consuming 643 GB of disk space and contained 10,641 files.  My customer asked the best way to clean the directory up and how to prevent this run away disk space situation from happening in the future.  This is very easy to accomplish through a feature that has been in the product for many years but is rarely activated.

The feature is enabled in the Configuration Document on the Diagnostics tab.  Set the parameter ‘Remove diagnostic files after a specified number of days‘ to “Yes” and then specify the number of days diagnostic files can remain on the server in the ‘Number of days to keep diagnostic files‘ parameter.  The default is 365 days.  This can be set to a lower value.  I would recommend keeping at least 30 days of diagnostic data, maybe even 60 to 90 days.

The graphic above shows adding this to the Default Configuration document so all Domino servers in the domain inherit this setting, however it can be set for a specific server as well by editing the Configuration Document for that one server.

It’s that simple!

A customer recently reported they were having issues adding people to the Certificate Authority (CA) process.  When they attempted to add people, they received the message “Cannot locate user certificate. Make sure server contains your certificate for encryption” as shown in the graphic below.

The customer had already done due diligence and verified their location document had the field “Home/mail server” set to the server where the ICL database resided, therefore meeting the requirement that the server listed in this field be in the same domain as the encrypting server for the CA.  Additionally, they ensured the field “Mail file location” was set to ‘Server’ and not ‘Local’ as that is also a requirement.

My first step was to check the status of the CA process on the server via tell ca status.  Everything was fine. I next examined the certifier being used, there were no issues found.  I now turned my focus to the ICL database.  The most recent IDStorage document had been last modified on July 12 2016.  That was over 3 months ago, so something else was amiss.

With this knowledge in hand, I clicked the ‘Advanced’ button on the Modify Certifier dialogue to attempt to replace the certifier ID and/or repair the ICL database.  This resulted in the same error, of user certificate could not be located.  The next step was to determine if there were any invalid Notes certificates in the person documents for the administrators that had been added to the CA process.  Once I had the list of CA administrators, I did a quick lookup of each of these users in the Domino directory.  What I discovered is that one of the CA administrators was no long in the directory.  This now explained the error message as every time the CA Process is updated, the certificates of all current users are verified.  Because this user was no longer in the directory, their certificate could not be located.

I had a copy of their Domino directory from earlier in the year stored locally, so I was able to copy the person document of that administrator back into their directory.  I then went into Modify the CA Process, removed the administrator that was no longer with the company, and the CA Process updated successfully as shown in the graphic below.

At this point I deleted the person document, using the delete key, not invoking the AdminP process using Delete Person as the user had previously been removed from the domain.

The moral of the story is to always remove an administrator from the CA process BEFORE deleting them from the Domino Directory as AdminP does not update the CA Process as part of the Delete Person process.

I had the opportunity to present at the Engage 2016 user group conference last week in Eindhoven.  I was extremely impressed on all levels.  Thank you to Theo Heselmans for the opportunity to speak at this event!  The session titles, abstracts and links to the presentations on SlideShare are below.

Adm02 Be a Domino Detective: Tackling Your Toughest Performance Problems

Link: http://www.slideshare.net/kkgreene/adm02-be-a-domino-detective-tackling-your-toughest-performance-problems-60128562

Abstract: Become a Domino performance investigator.  This session will teach you industry best practices for Domino performance optimization.  Learn how to take abstract symptoms like “Notes is slow” and break it down to a resolvable problem.  See the methodology and tricks involved to find the true culprit using tools such as semaphore timeouts, memory dumps and server monitoring.  Understand what impact running with obsolete tuning parameters can have on your environment. You will learn the best tips to implement along with do’s and don’ts for ensuring your Domino environment will perform optimally.

Adm07 The Health Check Extravaganza for Social and Collaboration Environments

Link: http://www.slideshare.net/kkgreene/adm07-the-health-check-extravaganza-for-ibm-social-and-collaboration-environments

Abstract: Are you concerned about your infrastructure being configured correctly? Do you have problems happen that you don’t know how to prevent? Do you think your servers might have room for improvement? Wonder no more. This session will show you what you need to be looking at to ensure your server environment is running as cleanly and efficiently as possible. You will learn what you need to be looking for in your server configuration, problems found at numerous customer environments and what steps should be taken to remedy the various situations covered in this session. Be preventative, not reactive! Performing a health check is one of the most economical ways to ensure your social and collaboration environments are running properly.

One of the critical things my company does for businesses is keeping their IBM Collaboration environments healthy.  We do this by reviewing the current environment and infrastructure, detailing our findings and making specific recommendations for improvement.  In doing this work, we have noticed patterns of findings.

At the Connect 2016 conference, myself and my colleague Luis Guirigay are presenting “The Health Check Extravaganza for Social and Collaboration Environments“.  We will be sharing with you our collective knowledge of our experiences in working with numerous customers in evaluating their collaboration environments.  We will share with you what you need to be looking at to ensure your server environments are running as cleanly and efficiently as possible.  Our goal is to be preventive, not reactive!

Performing a health check is one of the most economical ways to ensure your social and collaboration environments are running properly.  I hope to see you at our session today, Tuesday, February 2nd, at 1:15 PM in room Lake Highland.

One of our customers asked me to analyze the performance of the IBM i LPAR that hosts their primary Domino and Sametime servers as they had some concerns.  The LPAR hosts 5 Domino servers (Mail, Application, Administration, Sametime, and Dev/Test) along with the Sametime DB2, SSC, Proxy, and Meeting servers.

My analysis revealed quite high CPU utilization at times, in addition to faulting issues in the *BASE memory pool. The *BASE memory pool is pool 2, where all of the Domino and WAS-based Sametime servers run by default. Some of the Domino servers (Administration, Dev/Test, and Domino Sametime) had previously been moved to separate memory pools.

To determine which servers were causing the bulk of the faulting the *BASE pool, I created a query against the QAPMJOBL performance monitor database file.  The two top faulters were the Mail and Application servers.  I made a recommendation to move the Mail and Application servers to their own memory pool, allocating 36 GB of memory to each memory pool as a starting point.  I also recommended moving the Administration, Dev/Test, and Domino Sametime server back to the *BASE memory pool. This quite dramatically changed the memory allocations on the server.

The table below shows the memory allocations as they were when I performed my analysis.

This next table shows memory allocations after implementing my recommendations.

My memory tuning recommendations were implemented on October 25th.  The impact on CPU utilization was quite dramatic as shown in the graphic below.

Proper memory allocation is key for the best Domino performance!

Welcome to my new blog!

I had previously blogged on BleedYellow.com, however that site is now defunct, so I have moved my blog. I chose DominoDiva.com for my new blog as my blog handle on BleedYellow was DominoDiva.

I am very much looking forward to getting back into blogging!!