When working on a problem for a customer, it was noticed there were several temporary files on the mail server in the /mail directory. This customer’s Domino servers run on the IBM i, so there are multiple Domino servers on the system, a total of four. I decided to do a search to determine how many temporary files existed on the entire IBM i system and where they were located.
I used the following command in Qshell to search for all of the files that ended in .TMP:
find /*| grep “.*.TMP$” | xargs ls -l >TMPFiles_WithCreationData.txt
The resulting file contained the Permissions, Owner, Object Size, Month/Day/Year the file was created along with the Directory Location. Below is a sample of what the resulting data looked like:
Because Qshell is case sensitive, I also did a search for all files ending in .tmp. I imported the data from both searches into a spreadsheet and did some calculations, the results were quite interesting:
The oldest files dated back to 1996!
Almost all of the .TMP/.tmp files on the mail server were in the /mail directory
All but two of the .TMP/.tmp files on the application server were in the data directory
The total size of all of the .TMP/.tmp files was just under 1.5 TB!
After presenting the data to the customer, we received permission to schedule time to take the servers down, delete the temporary files, and restart the servers.
I used the rm command to do the deletions. Below are the steps to invoke QShell, change into the directory to perform the deletions in, issue the ls command to list out the files and then issue the rm command to delete the files.
In helping a customer who was having an issue with getting TOTP working, I came upon an interesting situation with their ID Vault. When issuing ‘show idvault’, the following error was displayed.
Invalid or nonexistent document: Vault replica list inconsistency for vault /ID_Vault
The really strange thing about this situation was the replica of the ID Vault was on both the primary and secondary server, however only the primary server was listed as a Vault Server in the ID Vault itself. Additionally, the Public Encryption Key for the RecoveryKeys document only had the primary server listed, the secondary server was missing.
It was unclear how this situation happened as the customer had created the vault replica on the secondary server via the Manage Vault interface and reported they received no errors performing this operation.
To debug and remedy the situation, I added the following debug variables to the notes.ini of both the primary and secondary servers.
CONSOLE_LOG_ENABLED = 1
DEBUG_IDV_CONNECT = 1
DEBUG_IDV_TRUSTCERT = 1
DEBUG_IDV_UPDATE = 1
I then deleted the secondary server via Manage Vault, ensured ‘show idvault’ no longer reported any errors, deleted the ID_Vault.nsf file on the secondary server. I then ended and restarted the secondary server and added the secondary server back in via Manage Vault.
Addition of the secondary server via Manage Vault failed with this error:
COULD NOT ADD SERVER02/ACME (FILE ALREADY EXISTS)
This was very strange as I had deleted the ID_Vault.nsf file on the secondary server and restarted it. In reviewing the console log, it became very clear what happened. The following errors show that Cluster Symmetry Repair had created the “missing” file on the secondary server.
In reviewing the cluster symmetry configuration, “All Folders” was selected for ‘Maintain symmetry’. As a temporary remediation until the Cluster Configuration could be thoroughly reviewed, I disabled repair on the ID_Vault.nsf database on both servers.
At this point, I was able to repeat the steps of removing the secondary server as a vault server, manually deleting the vault from the secondary server, restarting the server and then adding the secondary as a vault server via Manage Vault.
This remedied this situation and allowed the Manage Vault process to properly create the replica on the secondary server.
Moral of the story:
Make sure the IBM_ID_VAULT directory is excluded from Cluster Symmetry Repair!
Myself and a team of volunteers are creating an education offering for HCL Domino on Power Systems (IBMi and AIX). We are looking for your feedback in order to ensure we are creating the correct content. Please provide your input in this survey we have created. This is a joint collaboration between COMMON.org and HCL Software Academy.
The survey is open through Friday, February 11 2022.
Customers that have their Domino servers running on the IBM i commonly use the Backup Recovery and Media Services (BRMS) software to do their backups. BRMS allows for on-line backups, making it quite popular as there is no server down-time when saving data.
I recently received a question from a customer who wanted to verify their nightly BRMS backups were backing up their .NLO files. Below are the steps I used to do this verification.
The first step is locating the ‘Volume Identifier’ to use when searching with WRKMEDBRM to see what specifically is backed up. This is accomplished by entering the following:
go brms -> 10. Scheduling -> 1. Work with all BRM scheduled jobs -> Option 5 to Work With next to STRMNTBRM -> Option 8 to Work with last submission -> Option 4 to Work with spooled files
Option 5 to Display next to file QP1ARCY which is the backup recovery report
Position to where the Domino saves start by doing a search with ‘WRKDOMSVR’.
Locate the ‘Volume Identifier’ and also the ‘File Sequence’ that contains a large number of objects as that is where the .NLO files will be located. In the example below the ‘Volume Identifier’ is “CD1423” and the ‘Sequence Number’ is “164”. Note the large number of objects in this volume sequence: 22,511 saved objects.
Enter the WRKMEDBRM command specifying the Volume Identifier located in the QP1ARCY file.
WRKMEDBRM VOL(CD1423)
Select option 13. Display contents next to the Volume Serial that is listed.
Scroll to the Domino server where the .NLO files are located.
Take option 9. Work with saved objects next to each Domino server entry that matches the ‘File Sequence’ identified above. In our case next to File Sequence ‘164’.
Here we see the /DAOS directory and its contents listed as being backed up.
Take Option 9. Work with directory information on the ‘Work with Link Information’ screen.
Select Option 9. Work with Objects on the next screen, ‘Work with Directory Information’, to see the contents of the directory to verify the individual .NLO files are indeed being backed up.
As the Strategic Education Team Leader, one of my volunteer positions for the COMMON organization, it is always rewarding to come up with new education offerings. The NAViGATE conference is just that! When we as an education team came up with the idea for the conference, we discussed the realities of virtual conference overload. This allowed us to brainstorm and come up with a new format for this education offering.
All sessions are 20 minutes in length and are organized into consecutive blocks of sessions. At the end of each block there is a session called “Office Hours”. These “Office Hours” sessions allow attendees to ask questions about the sessions in the block and to have one-on-one time with the expert that delivered the bulk of the sessions in the education block.
I will be presenting a block of Domino and Sametime sessions that are followed by an “Office Hours” session. Below are the sessions in the block that I will be presenting:
Benefits of Upgrading and Installing Domino 10 and 11
Abstract: Domino 10 and 11 offer an extensive set of new features and functions. In this session, Steve Pitcher and Kim Greene will discuss some Domino history, features of HCL Domino 10, features of HCL Domino 11, Traveler, and how to upgrade.
Secure Instant Messaging without WebSphere
Abstract: The usage of corporate instant messaging has increased as companies have grappled with work from home due to the Covid-19 pandemic. In 2019 67% of companies utilized some form of instant messaging, in 2020 that increased to 81%. This session will introduce you to a secure instant messaging platform that is simple to setup and use, no more WebSphere!
Securing your Sametime 11.x Proxy and Community Servers
Abstract: Learn the key steps involved in securing your Sametime Community and Sametime Proxy servers to ensure all communication on this instant messaging platform is secure.
Office Hours: Sametime and Domino with Kim Greene
Abstract: Attend to ask your questions about Sametime and Domino. With Kim Greene, President, Kim Greene Consulting.
I will also be presenting the following session in the ‘Business and Personal Development’ block.
The Importance of Focus
Abstract: Focus is critical for accomplishing your goals. Whether it’s prioritizing work items, specking out a new project, designing a new application, or architecting new infrastructure, focus is key. Learn how to sharpen and enhance your focusing skills.
The NAViGATE conference allows for both in-person and virtual attendance. There are over 250 sessions in this conference. The great thing is if you are unable to attend a session, you can listen to it until early July. Don’t wait, sign up today for this high energy, action packed, fast paced education offering!! More details can be found here: https://www.common.org/columbus2021/home
One additional note, guest blogger Michelle Smith will be writing a series of blog posts detailing how to Secure your Sametime 11.x Proxy and Community Servers so check back soon!
HCL Enterprise Integrator (HEI), is more commonly known as Lotus Enterprise Integrator (LEI). You may also know it as IBM Enterprise Integrator (IEI). Despite the name changes, this product is just as amazing as it has always been. Regardless if Domino is one of the data stores or not, it synchronizes data between disparate systems. This means perhaps syncing data between DB2, Oracle, SQL Server, SAP, Domino, File Systems, … the list goes on. In addition this amazing product allows for virtual views and virtual fields when interfacing with Domino applications.
Enough reminiscing about functionality, what happens when you upgrade the product on the IBM i to version 11.0.1?
For those of us familiar with the product, we know the version of HEI needs to match the version of Domino. This does not change when upgrading to HEI 11.0.1. What does change is related to the Licensed Program Product for HEI when installing on the IBM i.
In prior versions, when installing HEI (well, technically LEI), the version of ‘LEI’ installed was removed and replaced with the new version. This is still true when installing HEI 11.0.1, but with a bit of a caveat. Even though there is a “success” message when installing HEI 11.0.1, when displaying the Licensed Program Products on the IBM i system, the big surprise is the product will still show as 5733LEI, not at all what one would expect!
As a result, the keys to ensuring LEI has been upgraded to version 11.0.1 is to validate the following:
Examine the contents of the ‘/qibm/proddata/lotus/domino110001’ directory. The following files should be present:
lei.lic
lei.pgm
leiact.pgm
leicsm.pgm
When starting the HEI server, the release level of HEI will be displayed and should show as being 11.0.1.
I will be co-presenting at the webinar “HCL Domino and Notes v12 are Coming!” on April 22 at 10:00 AM EDT / 4:00 PM CEST. This webinar is co-sponsored by HCL and panagenda. Barry Rosen will share details on what is coming in version 12, with a primary focus on the Notes client. He will also share details on what is coming in future releases.
Christoph Adler and myself will talk about upgrading the Notes client. Christoph will share details on how to automate the upgrade process. I will focus on the customer experience in upgrading the Notes client. Key topics we will cover include:
Auditing Existing Notes Client Deployments
Centralized Tracking and Reporting for Client Upgrades
Domino 10.0.1 has been available on the IBM i for a number of months now, however there have been no Fix Packs available for 10.0.1 on the platform to this point. That changes today, October 4 2019.
Many have been wondering where FPs for 10.0.1 on the IBM i have been. The reality is 10.0.1 for the IBM i has been 10.0.1.1 as the code load contains the fixes included in FP1. However, there have been no 10.0.1 FPs for the IBM i since, until today. This is due to issues HCL found in their internal testing that prevented them from releasing 10.0.1 FPs for the platform.
The issues are still being worked, so what HCL has done in order to get the FP4 code out to the IBM i community is create a full install for FP4. This means the prior 10.0.1 product needs to be deleted before FP4 can be installed. As is the case for the full product load, there are 3 savefiles associated with the installation.
QNOTES – *BASE of the 10.0.1 product
QNOTESAP – Option 1 which is the C API
QNOTESRL – Option 11 which is 10.0.1 (with FP4)
A couple of things to be aware of with FP4 being a full installation rather than just installation of a Fix Pack:
If you have Traveler installed with 10.0.1, you will need to reinstall Traveler after installation of FP4.
If you have any language packs installed, they will need to be reinstalled after installation of FP4.
For the official technote regarding FP4 for the IBM i refer to KB0069162.
Details on the installation of FP4 are in KB0069486.
It’s coming … just around the corner … can’t wait!!!!!!
One of the biggest questions I have been getting is which IBM i operating systems will support Domino 10.0.1. You will be very pleased with the answer 🙂 The releases supported are: 7.2, 7.3, and 7.4 (when it becomes available). The huge relief is that 7.2 is supported, whew!
In addition to the plethora of functionality that is provided in release 10.0, 10.0.1 brings some unique enhancements that are specific to the IBM i, let’s take a look at what these are.
Daylight Saving Time enhancement
This first one is long overdue. It’s one that is very subtle, until daylight saving time hits, then wham!! In Domino 10.0.1 on the IBM i, it will default to *TIMEZONE when you configure a new server. This means no more times being off by one hour when DST sets in because you forgot to add the DSTLAW= variable to the notes.ini when you configured the server.
Easier control over JVM used by Domino
Another enhancement to make administrator’s lives easier is the ability to change which JVM is in use for a Domino server. By default the 32-bit JVM is used for Domino on the IBM i. The process of changing from the 32-bit to the 64-bit JVM has been to add the JAVA_HOME variable to the notes.ini. While this doesn’t sound like a big deal, it can be a bit tricky. Below is the JAVA_HOME variable to convert a Domino server to use the 64-bit Java 8 JVM.
The main trick is case sensitivity, specifically with /QOpenSys. If you do not have the correct case for this portion of the path, it will not work. Domino 10.0.1 to the rescue! In this new release, there is a JVM version option available with both the Configure Domino Server (CFGDOMSVR) and Change Domino Server (CHGDOMSVR) commands.
MEMCHECK automatically collected in Domino 10.0.1
If you get into debug mode with your Domino servers, you will be elated that MEMCHECK is enabled by default on an IBM i Domino 10.0.1 server. You may be wondering why this is a big deal. Prior to Domino 10.0.1, this valuable debug data was not available unless you manually ran a memcheck on the IBM i, which very few customers know how to do.
Licensed Program Product (LPP) and Library changes
Other changes to be aware of are the licensed program product ID and library where the Domino 10.0.1 code resides. We are used to seeing 5733L85 and 5733LD9 for 8.5.3 and 9.0.1 respectively for the product ID. The 10.0.1 product will be 5733D10 with *BASE being the Domino 10.0 code and Option 1 being 10.0.1. We can quite easily guess that 10.0.2 will be Option 2.
The library where the Domino 10.0.1 code executables resides is probably the most noticeable difference. With libraries having a maximum length of 10 characters, the nomenclature of QDOMINO8xx and QDOMINO9xx no longer works with version 10.x. Therefore the library structure changes to QDOM100001 for version 10.0.1. The table below provides a summary of what the Domino releases and their respective LPP, library, and directory details look like for releases 8.5.3, 9.0.1 and 10.0.1.
Domino Release
LPP ID
Option
Program Library
Product Directory
8.5.3
5733L85
*BASE, 1, 13
QDOMINO853
/QIBM/ProdData/LOTUS/domino853
9.0.1
5733LD9
*BASE, 1, 11
QDOMINO901
/QIBM/ProdData/LOTUS/domino901
10.0.1
5733D10
*BASE, 1, 11
QDOM100001
/QIBM/ProdData/LOTUS/domino100001
When do we get all of this?!?
So now the big question remains, WHEN will Domino 10.0.1 be available on the IBM i?!?
I have some important discoveries to share on this front. In preparing for the Domino Technical University webcast I will be doing with Chris Adler on February 7th on Best Practices to Upgrading to Domino 10 (https://bit.ly/2BiMrVO), I found the following when reviewing the product documentation:
In addition when searching in Passport Advantage, I see this for Domino 10.0.1 for the IBM i.
When clicking the link to download it, this secret is revealed 🙂
I hope you are as excited as I am to get this phenomenal, feature rich, self-healing release available on the hardware platform that is most synonymous with Domino, the IBM i: it’s integrated, secure, scalable, and highly available.
Many of my customers, colleagues and friends in the community have been asking me when and “if” Domino 10 is going to be available on the IBM i. The rate of questioning increased after the announcement that HCL is going to purchase Domino, progressing beyond the current partnership agreement between IBM and HCL.
At first I was a bit perplexed why there was more concern with the purchasing agreement vs. the partnership agreement. I asked some questions and the answers became obvious. People’s assumption was that the IBM i would be supported by IBM because it owns the system and HCL does not. While I completely understand this assumption, reality couldn’t be farther from the truth.
HCL has listened to ALL customers of the Domino platform and has realized that IBM i customers are very committed to the product and many use it to provide key components of their businesses. In addition, HCL gets what Domino really is, an application development platform. They don’t see it as just an email, calendar, and contacts solutions which sadly is how the ones in IBM who decide where $$ go see it; rather HCL sees the significant advantage this low-code platform provides to companies to give them a competitive advantage.
HCL is fully committed to the IBM i platform, in ways we have not seen since the early 2000s. I am under NDA, so I cannot say exactly when Domino 10 will be available on the IBM i, what I can safely say is it will be in the first quarter of 2019.
Recent Comments