You are viewing perfthoughts

Tivoli Security Performance Thoughts
Explorations and ruminations on performance
Recent Entries 
Last week member of the ITIM development team published a package that enables monitoring of ITIM 5.1 with IBM Tivoli Monitoring 6.2.2. This compliments the package that enables monitoring of ITIM 4.6 with IBM Tivoli Monitoring 6.1 that has been available for several years.
I've posted an article on the ITIM Wiki titled Leveraging LDIFs and LDAP commands in test environments. This document discusses some of the tricks we use internally for testing ITIM that are likely to be useful for customer test environments as well.
The fifth edition of the ITIM 5.0 and 5.1 Performance Tuning Guide was released today. As promised, this release looks very different than the previous ones as it now matches the other ITIM documentation. I expect this change will take a bit of getting use to but I think it will provide a better overall experience -- particularly when the guide is fully integrated into the ITIM Information Center in a month or so.

Beyond the look-and-feel, the following content changes were updated:
  • Enabling content compression for the IBM HTTP Server - minor changes to the IBM HTTP Server configuration file
  • Configuring assembly line caching - updated to reflect the latest Dispatcher release
and the following content was added:
  • Improving the caching of static content served from the IBM HTTP Server - IBM HTTP Server configuration file additions to improve browser-side caching of static content
  • Configuring the number of concurrently running assembly lines - explanation of the GlobalRunALCount Dispatcher configuration parameter
  • Configuring LDAP connection pooling - configuration file change to improve overall performance for environments using SSL between ITIM and LDAP
  • Configuring reconciliation threads - explanation of the ITIM reconciliation threading model and how to tune the number of threads for your environment

Update: One user reported having problems printing the PDF but was able to work around the issue by skipping pages 1-2. We've not been able to reproduce the problem but try this workaround if you encounter it.
DB2 9.5 introduced a way to update the automatic maintenance policies via the command line instead of relying on the DB2 Control Center. This makes enabling automatic runstats on the ITIM and ITDS databases much easier as updating the policies is a required step of that process.

To get the current AUTO_RUNSTATS policy, connect to the database and use:
   db2 "call sysproc.automaint_get_policyfile( 'AUTO_RUNSTATS', 'AutoRunstats.xml')"
The AutoRunstats.xml file will be written to the $HOME/sqllib/tmp directory.

To set the AUTO_RUNSTATS policy, use:
   db2 "call sysproc.automaint_set_policyfile( 'AUTO_RUNSTATS', 'AutoRunstats.xml')"
The AutoRunstats.xml file location is relative to the $HOME/sqllib/tmp directory.

The following AutoRunstats.xml can be used for the ITIM DB:
<?xml version="1.0" encoding="UTF-8"?>

<DB2AutoRunstatsPolicy
xmlns="http://www.ibm.com/xmlns/prod/db2/autonomic/config" >
 <RunstatsTableScope>
  <FilterCondition>tabname not in ('SIBOWNER','SCHEDULED_MESSAGE','PROCESSDATA')</FilterCondition>
 </RunstatsTableScope>
</DB2AutoRunstatsPolicy>

And here's a version of the policy that can be used for the ITDS DB:
<?xml version="1.0" encoding="UTF-8"?>

<DB2AutoRunstatsPolicy
xmlns="http://www.ibm.com/xmlns/prod/db2/autonomic/config" >
 <RunstatsTableScope>
  <FilterCondition>tabname not in ('LDAP_ENTRY','LDAP_DESC','ERPARENT')</FilterCondition>
 </RunstatsTableScope>
</DB2AutoRunstatsPolicy>
1st-Jun-2010 07:27 am - Next ITIM Tuning Guide: Coming soon
I'm sure some of you are scratching your heads about when the next ITIM V5.x Tuning Guide is coming out. We usually release one every 6 months and we're coming up on a year since the last update. We had hoped to get one out at the end of May but the date has been pushed out to likely the end of June.

The Guide has become stable enough that after 7 years it was time to move it from a development-managed document to a document managed by our information development group. In doing so the document was migrated from Microsoft Word to DITA which took longer than expected. The next edition will have a distinctly different look-and-feel from the previous incarnations and will instead match the existing ITIM documentation -- including ready-inclusion into the ITIM Information Center.

The contents will still be maintained by Subject Matter Experts as it has in the past. This change in format will allow us to provide a better and more accessible document than we were able to do before.

Once the next edition is released I'll announce it here as usual along with the most important additions or changes, most all of which have already been discussed here previously.
As previously mentioned the most recent UnixLinux adapters improve overall reconciliation performance. In the lab we've been playing with other ways to further improve their performance. One way we've identified is a endpoint-side cache of the reconciliation data that is refreshed asynchronously from the reconciliation process. The downside of this approach is that the cache needs to be refreshed via code deployed on the endpoint either via remote execution of the script on the endpoint or crontab on the endpoint itself. Because of this limitation it is unlikely to be rolled into the GA adapters but it might be useful for specific endpoints (slow ones or large ones for instance) or for customers who have tight control over their endpoints and want the reconciliation performance boost.

The caching change is done in the UnixLinux reconciliation scripts included with the adapter. The change is to detect if a cache file exists and is "fresh enough". If it is fresh enough we return the cached content, appropriately filtered, and exit. If it isn't fresh enough it is deleted. The reconciliation data is then calculated, cached, and returned. "Fresh enough" can be whatever it is you want it to mean -- in the example below it's defined as '1 day old'. This would allow up to daily reconciliations and ensure that the data being returned is, at most, 24 hours old.

Here's a unified diff of the LinuxShadowPConnRes.sh script with the changes. Similar changes could easily be made for the other scripts.
# diff -u 5.1.5_adapter/LinuxShadowPConnRes.sh LinuxShadowPConnRes.sh-cache 
--- 5.1.5_adapter/LinuxShadowPConnRes.sh	2010-04-19 09:23:40.000000000 -0600
+++ LinuxShadowPConnRes.sh-cache	2010-05-11 15:27:22.000000000 -0600
@@ -40,6 +40,39 @@
 
 BANG="!"
 
+# use the cache if possible
+cache_file=/tmp/ITIM_POSIX_adapter_cache.txt
+generate_cache_file=false
+# check to see if the file exists and is bigger than 0 bytes
+if [ -s $cache_file ]; then
+   # check to the see if the cache is fresh enough. to do this we create an
+   # empty file with a specific timestamp, and use the -nt comparison operator
+   # to test for freshness.
+   cache_file_timetest=/tmp/ITIM_POSIX_adapter_cache.tmp
+
+   # fresh enough, in this case, is 1 day old (date -d'-1 day')
+   # this can be changed to any date supported by the date -d command
+   touch -t `date -d'-1 day' +"%Y%m%d%H%M.%S"` $cache_file_timetest
+
+   if [ $cache_file -nt $cache_file_timetest ]; then
+      rm $cache_file_timetest
+      # we're fresh enough, return contents appropriately filtered
+      cat $cache_file | $filter
+      exit 0
+   else
+      rm $cache_file_timetest
+      # cache isn't fresh enough, remove it
+      rm $cache_file
+   fi
+fi
+
+# only generate the cache if our filter is "everything"
+if [ "$1" = "grep -e :" ]; then
+   generate_cache_file=true
+   touch $cache_file
+   chmod 600 $cache_file
+fi
+
 #confirm that faillog is installed/configured
 is_faillog_configured=false
 $faillogcmd -u root > /dev/null 2>&1
@@ -161,6 +194,9 @@
 
 
    echo $oneline
+   if [ "$generate_cache_file" = "true" ]; then
+      echo $oneline >> $cache_file
+   fi
 done
 #reset the Internal Field Separator
 IFS=$OIFS
The modified script would replace the instance in the RMI Dispatcher as well as be the one run on the endpoint (hint: for all scripts the second argument is 'grep -e :' to return all users (ie: unfiltered) and the third is 'true' if using sudo or 'false' otherwise). The asynchronous run of the script would need to be done as the user doing the recon (ie: root or non-root as sudo) as the cache file is owned by that user and is only accessible to the owning user (chmod 600).

The nice thing about the caching changes as currently coded is that it can be used for all endpoints without negatively impacting performance but those endpoints that are updating the cache asynchronously will see a significant speed improvement.
The ITIM Performance Team has adopted the following model for publishing tuning guides for product versions reaching their end of support:
  • Tuning guides will not be updated once that product version reaches its end of support date.
  • Tuning guides will remain accessible for one year after the product version's end of support date. After this time it will be removed from the website.
IBM Tivoli Identity Manager 4.6 Express reaches it's end of support this week. The tuning guide for this version will remain accessible until April 2011 when it will be removed from the website.

This week also marks a year since IBM Tivoli Identity Manager 4.5.1 reached its end of support. As such the 4.5.1 tuning guide will no longer be available after the end of this week.
During some recent testing I discovered an interesting, and a bit unexpected, gotcha when doing reconciliations of LDAP endpoints. While reconciling an LDAP endpoint where no accounts had been modified since the last reconciliation, I discovered that ITIM was still detecting that the accounts were changed, albeit compliant, and doing an update to the ITIM LDAP for the account.

After a bit of digging I discovered that because I was binding as an administrative user to the LDAP endpoint (in my case cn=root), the userPassword attribute was being returned and because ITIM doesn't 'see' that attribute in its record for the account, it assumes it is changed and pushes the update to the ITIM LDAP. This is very non-optimal as while changed-but-compliant accounts result in less overhead than changed-and-noncompliant accounts, they still have more overhead than unchanged accounts. Updating the reconciliation schedule to exclude this attribute (listed as User Password in the selection box) yielded the expected results of all the accounts being recognized as unchanged.

If you're binding as an administrative user to an LDAP endpoint and using ITIM as an authoritative password source for that resource, ensure that the userPassword attribute is excluded from all reconciliation schedules for the best reconciliation performance. If you're binding as a non-administrative user this isn't necessary as the userPassword attribute isn't visible to these users.
23rd-Apr-2010 09:30 am - Improving ITIM performance over a WAN
Today I published an article on the ITIM wiki titled Improving IBM Tivoli Identity Manager performance over a WAN with a reverse proxy. This article was written with collaboration from Andrei Iakovlev, an IBM IT Security Architect as we worked on solving WAN performance architecturally for a client.

Suggestions and feedback about the article are welcome.
22nd-Apr-2010 08:06 am - ITIM reconciliation threads redux
After re-reading my last post I realize I did a good job of muddling together two distinct, but related, findings:
  • a single ITIM recon thread is capable of processing accounts faster than most adapters can return them
  • decreasing the number of recon threads when adapters return accounts faster than a single ITIM thread can process them can maintain current recon speeds and decrease the CPU used on the ITIM node
And if you think about it, the above makes sense: the CPU aspect is due to thread contention which is only possible if there is more work than a single thread can manage on its own.

The recommendation in both cases is the same: decrease the number of recon threads from 8 down to 4 or 2. If the adapter is returning accounts faster than ITIM can process them, this will decrease CPU utilization caused by thread contention. If a single ITIM thread can process the accounts faster than the adapter can return them, this will decrease the number of idle threads and no resources are consumed by the JVM to create, track, and destroy them.

An obviously related question is: if ITIM is waiting on most adapters to return accounts, can I get better overall recon throughput by increasing the number of concurrent recons that are running? The answer is a qualified yes based on resource constraints. We've done some preliminary testing in this area and expect to be able to make more solid recommendations soon.
This page was loaded Jul 24th 2014, 3:56 am GMT.