Recovering Disk Space by Deleting TMP files

When working on a problem for a customer, it was noticed there were several temporary files on the mail server in the /mail directory. This customer’s Domino servers run on the IBM i, so there are multiple Domino servers on the system, a total of four. I decided to do a search to determine how many temporary files existed on the entire IBM i system and where they were located.

I used the following command in Qshell to search for all of the files that ended in .TMP:

find /*| grep “.*.TMP$” | xargs ls -l >TMPFiles_WithCreationData.txt

The resulting file contained the Permissions, Owner, Object Size, Month/Day/Year the file was created along with the Directory Location. Below is a sample of what the resulting data looked like:

Because Qshell is case sensitive, I also did a search for all files ending in .tmp. I imported the data from both searches into a spreadsheet and did some calculations, the results were quite interesting:

1. The oldest files dated back to 1996!
2. Almost all of the .TMP/.tmp files on the mail server were in the /mail directory
3. All but two of the .TMP/.tmp files on the application server were in the data directory
4. The total size of all of the .TMP/.tmp files was just under 1.5 TB!

After presenting the data to the customer, we received permission to schedule time to take the servers down, delete the temporary files, and restart the servers.

I used the rm command to do the deletions. Below are the steps to invoke QShell, change into the directory to perform the deletions in, issue the ls command to list out the files and then issue the rm command to delete the files.

1. qsh
2. cd /notes/data/mail
3. ls *.tmp
4. ls *.TMP
5. rm *.tmp
6. rm *.TMP

Happy deleting!

ID Vault “Invalid or nonexistent document” error caused by Cluster Symmetry Repair

In helping a customer who was having an issue with getting TOTP working, I came upon an interesting situation with their ID Vault. When issuing ‘show idvault’, the following error was displayed.

Invalid or nonexistent document: Vault replica list inconsistency for vault /ID_Vault

The really strange thing about this situation was the replica of the ID Vault was on both the primary and secondary server, however only the primary server was listed as a Vault Server in the ID Vault itself. Additionally, the Public Encryption Key for the RecoveryKeys document only had the primary server listed, the secondary server was missing.

It was unclear how this situation happened as the customer had created the vault replica on the secondary server via the Manage Vault interface and reported they received no errors performing this operation.

To debug and remedy the situation, I added the following debug variables to the notes.ini of both the primary and secondary servers.

• CONSOLE_LOG_ENABLED = 1
• DEBUG_IDV_CONNECT = 1
• DEBUG_IDV_TRUSTCERT = 1
• DEBUG_IDV_UPDATE = 1

I then deleted the secondary server via Manage Vault, ensured ‘show idvault’ no longer reported any errors, deleted the ID_Vault.nsf file on the secondary server. I then ended and restarted the secondary server and added the secondary server back in via Manage Vault.

Addition of the secondary server via Manage Vault failed with this error:

COULD NOT ADD SERVER02/ACME (FILE ALREADY EXISTS)

This was very strange as I had deleted the ID_Vault.nsf file on the secondary server and restarted it. In reviewing the console log, it became very clear what happened. The following errors show that Cluster Symmetry Repair had created the “missing” file on the secondary server.

In reviewing the cluster symmetry configuration, “All Folders” was selected for ‘Maintain symmetry’. As a temporary remediation until the Cluster Configuration could be thoroughly reviewed, I disabled repair on the ID_Vault.nsf database on both servers.

At this point, I was able to repeat the steps of removing the secondary server as a vault server, manually deleting the vault from the secondary server, restarting the server and then adding the secondary as a vault server via Manage Vault.

This remedied this situation and allowed the Manage Vault process to properly create the replica on the secondary server.

Moral of the story:

Make sure the IBM_ID_VAULT directory is excluded from Cluster Symmetry Repair!

Requesting Your Input on Education Needed for Running Domino on Power Systems

Myself and a team of volunteers are creating an education offering for HCL Domino on Power Systems (IBMi and AIX). We are looking for your feedback in order to ensure we are creating the correct content. Please provide your input in this survey we have created. This is a joint collaboration between COMMON.org and HCL Software Academy.

The survey is open through Friday, February 11 2022.

Start the survey

Thank you in advance!

Is your Notes 11 client crashing repeatedly? Look at your Chrome settings…

One of our clients reported daily Notes Client crashes for some of their users after upgrading their Notes 9.x clients to 11.0.1 FP3 (and later FP4). We narrowed it down to a certain set of steps that would INTERMITTENTLY result in a crash.

Joe (I will call him Joe, mostly because that is his name but also because he is not the only Joe in this world) would do the following:

1. Edit an existing appointment in his calendar, or another shared calendar

2. Update some text in the description

3. Change the date to a future date

… before he could click “Save and Close” his Notes Client would sometimes crash.

Now, I will say Joe is one of the most patient people I have ever worked with, as this was very frustrating. Since the problem was intermittent, there were multiple iterations of analysis and remedying done, with a period of waiting time to see if the issue reoccurred.

An HCL Support case revealed that due to the presence of core_*.* files in the Support Data collected, there was something happening at the O/S level at the time of the crash. Their analysis also found over 80% of memory was being utilized at the time of the crashes.

What was using up all the memory? Task Manager consistently indicated Chrome was the largest consumer of memory.

After a bit of digging into Google Chrome settings, we found that hardware acceleration can sometimes help performance and sometimes hurt performance. We decided to try changing Joe’s current settings. He had these two settings on, so we turned them off and had him restart his laptop.

The result? It’s been almost 2 weeks and he has had no more Notes Client crashes! Joe is very pleased. His co-workers who had been having the same problem are no longer having Notes Client crashes since their Chrome settings were updated.

Bonus: Did you know that Chrome has its own Task Manager? SHIFT-ESC while in Chrome will show you the Chrome specific details.

Verifying if NLO Files are Being Backed up by BRMS

Customers that have their Domino servers running on the IBM i commonly use the Backup Recovery and Media Services (BRMS) software to do their backups. BRMS allows for on-line backups, making it quite popular as there is no server down-time when saving data.

I recently received a question from a customer who wanted to verify their nightly BRMS backups were backing up their .NLO files. Below are the steps I used to do this verification.

• The first step is locating the ‘Volume Identifier’ to use when searching with WRKMEDBRM to see what specifically is backed up. This is accomplished by entering the following:
  • go brms -> 10. Scheduling -> 1. Work with all BRM scheduled jobs -> Option 5 to Work With next to STRMNTBRM -> Option 8 to Work with last submission -> Option 4 to Work with spooled files
  • Option 5 to Display next to file QP1ARCY which is the backup recovery report
• Position to where the Domino saves start by doing a search with ‘WRKDOMSVR’.
• Locate the ‘Volume Identifier’ and also the ‘File Sequence’ that contains a large number of objects as that is where the .NLO files will be located. In the example below the ‘Volume Identifier’ is “CD1423” and the ‘Sequence Number’ is “164”. Note the large number of objects in this volume sequence: 22,511 saved objects.
  • 
• Enter the WRKMEDBRM command specifying the Volume Identifier located in the QP1ARCY file.
  • WRKMEDBRM VOL(CD1423)
• Select option 13. Display contents next to the Volume Serial that is listed.
• Scroll to the Domino server where the .NLO files are located.
• Take option 9. Work with saved objects next to each Domino server entry that matches the ‘File Sequence’ identified above.  In our case next to File Sequence ‘164’.
  • 
• Here we see the /DAOS directory and its contents listed as being backed up.
  • 
• Take Option 9. Work with directory information on the ‘Work with Link Information’ screen.
• Select Option 9. Work with Objects on the next screen, ‘Work with Directory Information’, to see the contents of the directory to verify the individual .NLO files are indeed being backed up.
  • 

Happy BRMS Backup Verification!

Excited to be Speaking at the NAViGATE Conference

As the Strategic Education Team Leader, one of my volunteer positions for the COMMON organization, it is always rewarding to come up with new education offerings. The NAViGATE conference is just that! When we as an education team came up with the idea for the conference, we discussed the realities of virtual conference overload.  This allowed us to brainstorm and come up with a new format for this education offering.

All sessions are 20 minutes in length and are organized into consecutive blocks of sessions.  At the end of each block there is a session called “Office Hours”.  These “Office Hours” sessions allow attendees to ask questions about the sessions in the block and to have one-on-one time with the expert that delivered the bulk of the sessions in the education block.

I will be presenting a block of Domino and Sametime sessions that are followed by an “Office Hours” session.  Below are the sessions in the block that I will be presenting:

Benefits of Upgrading and Installing Domino 10 and 11

Secure Instant Messaging without WebSphere

Abstract: The usage of corporate instant messaging has increased as companies have grappled with work from home due to the Covid-19 pandemic.  In 2019 67% of companies utilized some form of instant messaging, in 2020 that increased to 81%.  This session will introduce you to a secure instant messaging platform that is simple to setup and use, no more WebSphere!

Securing your Sametime 11.x Proxy and Community Servers

Abstract: Learn the key steps involved in securing your Sametime Community and Sametime Proxy servers to ensure all communication on this instant messaging platform is secure.

Office Hours: Sametime and Domino with Kim Greene

Abstract: Attend to ask your questions about Sametime and Domino.  With Kim Greene, President, Kim Greene Consulting.

I will also be presenting the following session in the ‘Business and Personal Development’ block.

The NAViGATE conference allows for both in-person and virtual attendance. There are over 250 sessions in this conference. The great thing is if you are unable to attend a session, you can listen to it until early July.  Don’t wait, sign up today for this high energy, action packed, fast paced education offering!! More details can be found here: https://www.common.org/columbus2021/home

One additional note, guest blogger Michelle Smith will be writing a series of blog posts detailing how to Secure your Sametime 11.x Proxy and Community Servers so check back soon!