Self-Healing Capabilities of Domino 10 presented at CollabSphere

I did a session at CollabSphere on the Self-Healing Capabilities of Domino 10. The functionality added into Domino 10 for maintaining a healthy Domino environment is absolutely amazing.  My presentation covers 5 issues that plague administrators on a day-to-day basis, how these issues are handled today and how these issues are remedied in Domino 10.  The 5 issues I covered are:

• Issue 1: Missing replicas and NLOs
• Issue 2: Corrupt NSFs and NLOs
• Issue 3: Missing documents
• Issue 4: Critical views out of date
• Issue 5: Who Deleted my documents

If you utilize clustering in Domino, you will be applauding Automatic Cluster Database Symmetry.  This functionality ensures both NSFs and NLOs are kept in sync across cluster mates.  I have never worked with a customer environment that didn’t have missing NSFs on their cluster mates and I have found when DAOS is being used, inevitably there are at least a handful of NLOs missing on one or more servers.  Automatic Cluster Database Symmetry ensures these discrepancies are repaired with no administrator intervention required once things are setup.

What’s even more cool is that if NLOs are encrypted with the server’s ID, the NLOs are decrypted with the source server’s ID, repaired/replicated, and then encrypted with the target server’s ID, provided NLO encryption is enabled (which is the default setting).

There is a new Cluster Configuration document that allows administrators to select which cluster to work with.  Based on the chosen cluster, the Symmetrical Cluster tab of the new configuration document allows you to select which folders should be monitored by the server to maintain symmetry on.  This allows you to keep both NSF and DAOS content synchronized across a cluster.

I have uploaded the presentation to SlideShare: https://www.slideshare.net/KGCI/self-healing-capabilities-of-domino-10-108188426

I would like to thank the HCL Development team, in particular Gary Rheaume, for all of their input on the content of the presentation.  Gary was awesome and created two videos I was able to use during my presentation as demos.  The great news is both of these videos are now available on YouTube!

The first demo is of Automatic Cluster Database Symmetry and can be found here: https://www.youtube.com/watch?v=8q-AWayRxwI

The second demo is of Automatic Database Repair and can be found here: https://www.youtube.com/watch?v=IKYLJWpEWBo

I will be creating additional blog posts regarding these amazing features coming in Domino 10.

Preventing Meltdown and Spectre on IBM i

To properly protect your IBM i systems, you need to install patches to both the OS and the firmware to address the Spectre and Meltdown security exposures.  Below are the required fixes depending on your OS level:

Release 7.1 – MF64553
Release 7.2 – MF64552
Release 7.3 – MF64551

Details can be found here: http://www-01.ibm.com/support/docview.wss?uid=nas8N1022433

The OS fix can be applied independently of the firmeware fix.  Keep in mind that BOTH the OS and firmware fixes are required to mitigate the security vulnerabilities presented by Spectre and Meltdown.

The firmware update can be obtained from FixCentral.  Download one of the following depending on your Power Systems firmware:

770.91 (01AL770_120_032, 01AM770_120_032)
780.81 (01AM780_094_040)
783.51 (01AF783_039_021)
860.42 (01SV860_138_056, 01SC860_138_056)

Details on the firmware updates can be found here: http://www-01.ibm.com/support/docview.wss?uid=isg3T1026811

Installing FP9 on IBM i is Quite Different and Very SLOW

If you are planning to install FP9, there are a few things you need to be aware of:

1. The installation process is quite different from what you are used to for installing FPs on the IBM i
2. It’s significantly slower, as in hours, so plan accordingly

Installation process

The installation process is a fair bit different than installing previous FPs.  You no longer just download the savefile, FTP it so the server and then load and apply the associated PTF.  Instead, you download the savefile, FTP it to the server and then things get a fair bit different.

Step 1.

The first step after downloading the savefile, which is called KITFP9, is to restore the contents of the savefile. This savefile contains two savefiles:

 

 

Make sure you leave the default of restoring the objects into the saved library of QGPL rather than specifying a library of your own, because the install script that is used to do the installation assumes the savefile for FP9 (QFP69019) is in library QGPL.

The restore command provided is:

RSTOBJ OBJ(*ALL) SAVLIB(QGPL) DEV(*SAVF) SAVF(QGPL/KITFP9)

 

Step 2.

The next step is to restore the console installer from savefile QCLINSTF.

RST DEV(‘/QSYS.LIB/QGPL.LIB/QCSLINSTF.FILE’) OBJ((‘/HOME/QNOTES’ *INCLUDE *SAME)) CRTPRNDIR(*YES) PRNDIROWN(QNOTES)

This will restore 81 files into /home/qnotes.

The first level is the console_install directory which contains the install script, install.sh, along with the LAP directory.

 

 

 

The Readme notes to NOT restore the contents into /QIBM/PRODATA/LOTUS/DOMINO901 as this is the directory where the Domino 9.0.1 executables are stored, so either leave the default of /home/qnotes or specify another directory you know is not in use by other applications.

Step 3.

As normal, end all active Domino servers on the system or LPAR that are running Domino 9.0.1.

Step 4.

This is where things get a bit different.  Rather than just issuing LODPTF and APYPTF, instead, you invoke QShell and run the installation script that is in the /home/qnotes/console_install directory. So the steps become:

QSH

cd /home/qnotes/console_install

./install.sh

The installer at this point will issue a LODPTF and APYPTF, remember it will bomb out unless the QFP69019 savefile is in library QGPL. I haven’t had time to work with modifying the install script and testing to see if things work okay with the having the savefile in another library, so for now stick with QGPL.

Step 5.

You will next be presented with the following screen regarding the license agreement.

                                                        QSH Command Entry

This document includes License Information documents below
for multiple Programs. Each License Information document
identifies the Program(s) to which it applies. Only those
License Information documents for the Program(s) for which
Licensee has acquired entitlements apply.

  ==============================================

LICENSE INFORMATION

The Programs listed below are licensed under the following

Press Enter to continue viewing the license agreement, or
enter “1” to accept the agreement, “2” to decline it, “3”
to print it, or “99” to go back to the previous screen.

You will want to type in 1 and press Enter to continue.

Step 6.

Wait, and wait, and wait, and wait, …. yes continue to wait, because it’s going to be a LONG, LONG time before the installation completes.  More on that in the Installation performance section below.

Step 7.

Once the installation finally finishes, load the most recent Interim Fix or custom hotfix if you have one from IBM.

In this case I installed IF2, so the commands were:

LODPTF LICPGM(5733LD9) DEV(*SAVF) SELECT(L605552) SAVF(GREENK/QL605552)

APYPTF LICPGM(5733LD9) SELECT(L605552)

Step 8.

Start the Domino servers and test.

Installation performance

The Readme states this: Domino 9.0.1 FP includes Dojo version upgrade with thousands of stream
files, need to plan for additional time required to copy these stream files to
Domino 9.0.1 product data and each server data folders during the installation
process.

The Readme for FP8 had this same statement, and the process was fairly quick, it is NOT for FP9, so what’s different?

From what I have observed, the big difference is how they handle copying the thousands of stream files.  In FP9, for every stream file that is copied, there are 3 messages that are generated and written to QShell in addition to be written to a spooled file:

1. CPCA087 which acknowledges the location the object was copied from and written to.
2. The first CPC221B does a Change Authority on the object that was copied.
3. The second CPC221B does a Change Owner on the object that was copied.

Below is an example for one stream file.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Not only are these 3 messages generated for the THOUSANDS of stream files it copies, the files are copied to every Domino server installed on that system or LPAR!!! None of this is done in parallel, the files are copied to the first server, then the next, and the next …  No wonder it takes FOREVER!

The system this installation was done on is an IBM Power System S814 with 3.72 GHz chip speed, 68 GB of memory and over 10 Terabytes of disk.  Needless to say there were ample resources available on this system to handle the workload of the installation. There are 4 Domino servers installed on this LPAR.

Over 3 hours after I started the installation of FP9, it finished.  This is unheard of.  When I installed FP8 on this system, it was done in around 10 minutes.

I generated a spooled file once the installation completed (pressed F6 in QShell).

The spooled file that was generated was 3,515 pages in length, and only contains output from the timeframe of 17:03:17 – 17:04:35.

You think someone forgot to turn off the debug code before this was shipped?!?

This problem has been reported to IBM with hopes the performance can be significantly improved.

Want Verse on Prem & Connections 6 on IBM i and AIX? Then Vote!

I have spent a fair amount of time this past year working with various groups within IBM to get Verse on Premises (VoP) supported on the IBM i. The net of these discussions is in order to get Verse on Premises supported on the IBM i or AIX platforms, Docker will be required. Docker is a technology that allows for applications to be created, deployed and run using containers.  By using Docker, applications and services can be built and deployed on any operating system that supports Docker.

In addition to Verse on Premises needing Docker, Connection 6 (PINK) will also require Docker.  In order to get these two collaboration solutions supported on IBM i and AIX, I have created two Requests For Enhancements (RFEs):

• Docker on IBM i, RFE # 112795
• Docker on AIX, RFE # 112797

I know both the IBM i and AIX communities want to have VoP and Connections PINK supported on their respective platforms.  The key is that the port of Docker to the IBM i will start with the port of Docker on AIX, so I am asking that you please vote for both of these RFEs to ensure both platforms become Docker enabled.

In addition to voting for the RFEs, adding comments to the RFEs will be very helpful to IBM in evaluating the importance of having Docker supported on these platforms.  By sharing why these platforms are important to you and how these collaboration solutions will benefit your business, IBM will have more knowledge when having discussions about these RFEs.

Please vote!!

 

My MongoDB Presentations at MWLUG

My presentations at MWLUG this year were a bit different from what I have presented at this conference in the past.  I submitted IBM Domino sessions for the Best Practices and System Administration tracks and MongoDB sessions for the Innovation track.  My session abstracts that were accepted were my two MongoDB sessions. I have been working with MongoDB about 6 months now and have found I have the same excitement and passion about MongoDB that I encountered when I first started working with IBM Domino over 20 years ago.  I find myself thinking “This is really cool stuff!”.

I didn’t know what to expect as far as turnout for the presentations. My first session, IN103 MongoDB – What You Need to Know, was a full house!  I am approximating there were about 70 people that attended this presentation.  I covered what MongoDB is, explained it’s multi-model database architecture and expressive query language in addition to other MongoDB basics.  I also explained how MongoDB is similar yet different from relational database systems, made comparisons between IBM Domino and MongoDB and how and why customers are using MongoDB.

My second session, IN106 MongoDB Performance, talked to the performance advantages of MongoDB over other data stores and delved into key factors for hardware requirements, and decisions that need to be made when choosing a MongoDB database type. I also covered what sharding is and how to choose the type of sharding that is best for a specific environment. along with critical elements for designing a schema that will perform in addition to the different types of and the importance of indexes.

Both of these presentations are on slideshare:

https://www.slideshare.net/KGCI/in103-mongodb-what-you-need-to-know

https://www.slideshare.net/KGCI/in106-performance-with-mongodb

My work and travel schedule has been fast and furious since less than 48 hours after MWLUG ended so my apologies for not getting these presentations made public until now.  I hope you enjoy them!

Domino on the IBM i (iSeries) IS supported!

I received an email today from one of my customers inquiring about this blog post: http://domino.elfworld.org/ibm-connect-2017-3-i-ve-seen-the-future-of-domino-and-it-is-sapho/

Their question to me was:

“Just a quick question, I was reading posts about Connection 2017 and came across this about halfway down.

Other news:
No more Notes client for Linux beyond 9.0.1 FP 7
32 bit droppet for AIX and Linux servers
No more Domino for iSeries

Is this true?”

I was at IBM Connect and had not heard this message and had previously been told that Domino on the IBM i would be supported along with all other currently supported platforms.  As a result, I reached out to Barry Rosen, the Offering Manager for IBM Collaboration Solutions.  He confirmed that Domino on the IBM i IS indeed supported.  He also posted the following comment on the blog post (thank you Barry!)

and the blog post has been updated to remove this statement (thank you Hogne).

The IBM Domino roadmap https://www.ibm.com/blogs/social-business/2016/09/12/ibm-notes-domino-v9-extends-support/ states “IBM Notes and Domino V9.0.1 is extending support through at least 2021”.  In my correspondences with Barry last October, I also received this statement in an email from Mr. Rosen that clearly states Domino on the IBM i is supported:

“Domino 9.0.1 on IBM i will be supported through at least 2021.  All current supported platforms will be supported through at least 2021.”

As I told my customer, rest assured, Domino on IBM i is fully supported, along with all feature packs.

Protecting your Domino servers from the clickjacking hack

There is a hack called clickjacking that can happen on web servers, including Domino.  Here are the details on how clickjacking can impact web sites.

An attacker performs a clickjacking attack by creating a site on the Internet, which contains inline frames (iframes) that can display content from the application.  The attacker sets the malicious iframes as invisible and places them on top of a commonly clicked link or icon found on the webpage.  Using JavaScript-based functions and other techniques, the attacker can force the authenticated user to click on and unknowingly execute target application functions. This exploit could control user’s actions without their knowledge and could potentially enable an attacker to expose confidential information or impersonate users.

For example let’s say users connect to the mail server via the URL https://mail.companyxyz.com.
This site can be included on a webpage with an iframe containing the following  <iframe src=”https://mail.companyxyz.com/” width=”500″ height=”500″></iframe>

The way you mediate this hack depends on the release level of the Domino server.

For any servers running 9.0.1 FP6 or higher, the following notes.ini variable can be set.  It just requires an end and restart of HTTP for this change to take effect.

HTTPAdditionalRespHeader=X-Frame-Options: SAMEORIGIN

For servers running earlier versions of Domino, those servers can be switched to use Internet Sites documents and then a Web Site Rule can be created that specifies a custom header with the x-frame-options header set to SAMEORIGIN.

If you haven’t enabled your server to use Internet Sites, edit the server document and specify “Enabled” for field ‘Load Internet configurations from Server\Internet Sites documents’.

Next create a Web Internet Site document, specifying the values appropriate for your site.  In the Web Site document, click Web Site -> Create Rule, select “HTTP response headers” for the ‘Type of rule’.  Under ‘Custom headers’, enter “X-Frame-Options” for the Name and “SAMEORIGIN” for Value and place a checkmark next to “Override“.

Whether you have enabled the notes.ini variable on a 9.0.1 FP6 or higher server or enabled the capability through a Web Site Rule in an Internet Site document, end and restart the HTTP task for prevention of clickjacking on your Domino server to be enabled.

Here is a technote for reference: http://www-01.ibm.com/support/docview.wss?uid=swg21568598

Keeping Domino Servers from Consuming Unnecessary Disk Space

I had a customer contact me recently asking about identifying why one of their IBM i LPARs had so much additional disk space consumed compared to another LPAR that hosted the Domino cluster mates.  The primary server hosted 4 Domino servers and the secondary LPAR hosted only 2 Domino servers.  The two additional servers on the primary LPAR were both Sametime Community servers, which typically don’t consume much disk space.

I used the disk usage utility to collect information on how much disk space each Domino server consumed.  The output is in the table below.

The Domino servers on the primary LPAR were consuming 843.34 GB more disk space than the servers on the secondary LPAR.  The two Sametime servers were a minuscule portion of this  discrepancy.  There were some databases on the primary servers (MAIL03 and COLLAB03) that were not on the cluster mates, however those databases only accounted for about 150 GB of space.  So what was causing such a disparity in the amount of space the primary servers were consuming compared to the secondary servers?

The culprit turned out to be the data in the IBM_TECHNICAL_SUPPORT subdirectory of the MAIL03 server.  This subdirectory alone was consuming 643 GB of disk space and contained 10,641 files.  My customer asked the best way to clean the directory up and how to prevent this run away disk space situation from happening in the future.  This is very easy to accomplish through a feature that has been in the product for many years but is rarely activated.

The feature is enabled in the Configuration Document on the Diagnostics tab.  Set the parameter ‘Remove diagnostic files after a specified number of days‘ to “Yes” and then specify the number of days diagnostic files can remain on the server in the ‘Number of days to keep diagnostic files‘ parameter.  The default is 365 days.  This can be set to a lower value.  I would recommend keeping at least 30 days of diagnostic data, maybe even 60 to 90 days.

The graphic above shows adding this to the Default Configuration document so all Domino servers in the domain inherit this setting, however it can be set for a specific server as well by editing the Configuration Document for that one server.

It’s that simple!

Remove Administrators from the CA Process BEFORE deleting them from the Domino Directory

A customer recently reported they were having issues adding people to the Certificate Authority (CA) process.  When they attempted to add people, they received the message “Cannot locate user certificate. Make sure server contains your certificate for encryption” as shown in the graphic below.

The customer had already done due diligence and verified their location document had the field “Home/mail server” set to the server where the ICL database resided, therefore meeting the requirement that the server listed in this field be in the same domain as the encrypting server for the CA.  Additionally, they ensured the field “Mail file location” was set to ‘Server’ and not ‘Local’ as that is also a requirement.

My first step was to check the status of the CA process on the server via tell ca status.  Everything was fine. I next examined the certifier being used, there were no issues found.  I now turned my focus to the ICL database.  The most recent IDStorage document had been last modified on July 12 2016.  That was over 3 months ago, so something else was amiss.

With this knowledge in hand, I clicked the ‘Advanced’ button on the Modify Certifier dialogue to attempt to replace the certifier ID and/or repair the ICL database.  This resulted in the same error, of user certificate could not be located.  The next step was to determine if there were any invalid Notes certificates in the person documents for the administrators that had been added to the CA process.  Once I had the list of CA administrators, I did a quick lookup of each of these users in the Domino directory.  What I discovered is that one of the CA administrators was no long in the directory.  This now explained the error message as every time the CA Process is updated, the certificates of all current users are verified.  Because this user was no longer in the directory, their certificate could not be located.

I had a copy of their Domino directory from earlier in the year stored locally, so I was able to copy the person document of that administrator back into their directory.  I then went into Modify the CA Process, removed the administrator that was no longer with the company, and the CA Process updated successfully as shown in the graphic below.

At this point I deleted the person document, using the delete key, not invoking the AdminP process using Delete Person as the user had previously been removed from the domain.

The moral of the story is to always remove an administrator from the CA process BEFORE deleting them from the Domino Directory as AdminP does not update the CA Process as part of the Delete Person process.

Engage 2016 User Group Conference

I had the opportunity to present at the Engage 2016 user group conference last week in Eindhoven.  I was extremely impressed on all levels.  Thank you to Theo Heselmans for the opportunity to speak at this event!  The session titles, abstracts and links to the presentations on SlideShare are below.

 

Adm02 Be a Domino Detective: Tackling Your Toughest Performance Problems

Link: http://www.slideshare.net/kkgreene/adm02-be-a-domino-detective-tackling-your-toughest-performance-problems-60128562

Abstract: Become a Domino performance investigator.  This session will teach you industry best practices for Domino performance optimization.  Learn how to take abstract symptoms like “Notes is slow” and break it down to a resolvable problem.  See the methodology and tricks involved to find the true culprit using tools such as semaphore timeouts, memory dumps and server monitoring.  Understand what impact running with obsolete tuning parameters can have on your environment. You will learn the best tips to implement along with do’s and don’ts for ensuring your Domino environment will perform optimally.

 

Adm07 The Health Check Extravaganza for Social and Collaboration Environments

Link: http://www.slideshare.net/kkgreene/adm07-the-health-check-extravaganza-for-ibm-social-and-collaboration-environments

Abstract: Are you concerned about your infrastructure being configured correctly? Do you have problems happen that you don’t know how to prevent? Do you think your servers might have room for improvement? Wonder no more. This session will show you what you need to be looking at to ensure your server environment is running as cleanly and efficiently as possible. You will learn what you need to be looking for in your server configuration, problems found at numerous customer environments and what steps should be taken to remedy the various situations covered in this session. Be preventative, not reactive! Performing a health check is one of the most economical ways to ensure your social and collaboration environments are running properly.