Shadow Caching

Last modified 26 Sep 2024 20:37 +02:00
Attribute caching feature
This page is an introduction to Attribute caching midPoint feature. Please see the feature page for more details.

MidPoint usually works with fresh data. When midPoint needs to get data about an account then the data will be retrieved on-demand from the resource. This is usually the best method. But there are cases when this approach is problematic. There may be resources that are often down. There may be resources that are very slow. Yet others may be costly or limited regarding accessing them, typically when in cloud. Therefore, midPoint has an ability to cache the values of resource objects and use them instead of retrieving them from the resource.

Moreover, there is another benefit of having the data cached: instant reporting. With the relevant parts of data being cached right in the repository, one can write reports against them, usually allowing for more complex queries and providing faster execution times.

How It Works

The only supported caching method is passive caching: Caches are maintained with minimal impact on normal operations. Generally the data are cached only if they are retrieved for other reasons. There is no read-ahead. The writes are always going to the resource (synchronously): read-through, write-through. There is no cache eviction (but old information is overwritten if newer information is available).

If caching is turned on, which is by default, then midPoint will cache the data in Shadow Objects. The caches are build gradually as midPoint reads the objects. If you need to populate the caches with information you have to make an operation that is searching all the objects. E.g. reconciliation or similar operation should do the trick. See Refreshing the Cache below.

Configuring Caching

The caching can be enabled or disabled at the level of a resource, object class, object type, or even an individual item (attribute, association, credentials, and so on).

An Example

For a quick overview, let us consider the following example. (The complete description is below.)

Listing 1: An example of custom caching configuration
<resource>
    ...
    <schemaHandling>
        <objectType>
            <kind>account</kind>
            <intent>default</intent>
            ...
            <attribute>
                <ref>ri:jpegPhoto</ref>
                <cached>false</cached> (4)
            </attribute>
            ...
            <caching>
                <cachingStrategy>passive</cachingStrategy> (1)
                <scope>
                    <attributes>all</attributes> (2)
                    <associations>none</associations> (3)
                </scope>
                <timeToLive>P1D</timeToLive>
                <defaultCacheUse>useCachedOrFresh</defaultCacheUse>
            </caching>
        </objectType>
    </schemaHandling>
    ...
    <caching>
        <cachingStrategy>none</cachingStrategy> (5)
    </caching>
    ...
</resource>
1 Enables the caching for account/default object type.
2 Enables the caching of all attributes.
3 Disables the caching of associations.
4 Overrides the caching for a specific attribute.
5 Disables the caching for all other objects.

Shortly speaking, the caching is disabled for the resource as a whole, except for the account/default type, for which it is enabled: Cached are all attributes except for jpegPhoto, credentials, activation, but no associations.

Configuration Details

At the level of resource, object class, or object type definition, the caching configuration item can be provided. It has the following sub-items:

Table 1. The caching configuration item content
Item Description Default

cachingStrategy

The overall switch that turns the caching on and off. It can have a value of none (no caching) and passive (passive caching as described above).

passive

scope

Scope of the caching (see below).

see below

timeToLive

For how long we consider the cached data as fresh.

P1D (or unlimited for caching-only read capability)

defaultCacheUse

How is the cache used for midPoint operations.

TODO reconsider cachedOrFresh (or fresh for caching-only read capability)
Table 2. The scope configuration item content
Item Description Default

attributes

Scope of the caching for simple attributes (i.e., not reference ones)

defined (or all for caching-only read capability)

associations

Scope of the caching for associations (i.e., reference attributes).

all

activation

Scope of the caching for the activation information.

all

auxiliaryObjectClasses

Scope of the caching for the auxiliary object classes.

all

credentials

Scope of the caching for credentials.

all (or none for caching-only read capability)
Table 3. The values of the scope configuration items for simple attributes
Value Description

none

No items of given kind will be cached.

defined

Only attributes defined (refined) in the schemaHandling section will be cached.

mapped

Only mapped items of given kind, i.e., those that have any mapping defined right in the object type, either inbound or outbound, will be cached.

all

All items of given kind will be cached.
Table 4. The values of the scope configuration items for other items (associations, activation, credentials, and so on)
Value Description

none

The data are not cached.

all

The data are cached.

Exceptions (both positive and negative) to the scope can be defined by using cached boolean property present for individual attributes and associations.

Configuring Cache Use

Even with caching turned on, the data may or may not be used for regular midPoint operation. For example, import from a resource may run against the actual resource data, or against the cached shadows in the repository. In a similar way, user recomputation may also use the actual or cached data. When someone opens a user projection in GUI, the cached or actual data can be displayed. And so on.

Use of Cached Data for Data Processing

This section describes the use of cached data for data processing in midPoint.

This primarily means providing source data for inbound mappings. But it also means providing the idea of what data are on resources when outbound mappings are concerned, e.g., for the correct application of weak and strong mappings.

What it does not cover, though, is the primary input of synchronization tasks, like import and reconciliation. It is taken always from the resource. (Unless turned off by an experimental setting described in Use of Cached Data for Import and Reconciliation Tasks section below.)

So, the data processing is driven by the defaultCacheUse configuration property. It can have the following values:

Value Description

useFresh

When we need the data from the shadow, we will fetch it from the resource. This is the same behavior as it was before midPoint 4.9.

useCachedOrFresh

When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will fetch them from the resource.

useCachedOrIgnore

When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will skip the respective part of the processing (e.g., a mapping).

Experimental.

useCachedOrFail

When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will fail.

Experimental.

For given task, the default behavior can be overridden using cachedShadowsUse model execution option (if that task does support specifying these options). However, this setting is currently experimental.

Use of Cached Data for Import and Reconciliation Tasks

The import and reconciliation tasks fetch their primary input, i.e., resource objects, right from the resource they are running against. It has a benefit of consistently update the cache for these objects.

There may be situations, though, when you’d need to avoid contacting the resource, and take data from the cache. It can be attained by specifying noFetch option, like this:

Listing 2: An example of custom caching configuration
<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
    oid="007c5ef2-3d1f-4688-a799-b735bbb9d934">
    <name>reconcile-hr-persons</name>
    <executionState>runnable</executionState>
    <activity>
        <work>
            <reconciliation>
                <resourceObjects>
                    <resourceRef oid="c37ff87e-42f1-46d2-8c6f-36c780cd1193"/>
                    <kind>account</kind>
                    <intent>person</intent>
                    <searchOptions>
                        <option>
                            <options>
                                <noFetch>true</noFetch> (1)
                            </options>
                        </option>
                    </searchOptions>
                </resourceObjects>
            </reconciliation>
        </work>
    </activity>
</task>
1 Specifies that the resource should not be contacted.
For the "remaining shadows" activity of reconciliation, where shadows that are presumably dead, are reconciled, we always go to the resource. If you need this behavior to be avoided, just turn off that activity.

This feature is experimental.

Refreshing the Cache

The best way of refreshing the cache is to run an import or reconciliation task. However, if you want to avoid the cost of data processing inherent in these tasks, you can use the following alternative:

Listing 3: A task that just refreshes the shadow cache
<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
      oid="45012c3e-3ce5-46ed-8d27-8648d2cbbca0">
    <name>Reload objects on HR resource</name>
    <ownerRef oid="00000000-0000-0000-0000-000000000002" type="UserType"/>
    <executionState>runnable</executionState>
    <activity>
        <work>
            <import>
                <resourceObjects>
                    <resourceRef oid="42a11d38-afbb-4f0e-8aea-c848db8ba0ab"/> <!-- HR -->
                    <kind>account</kind>
                    <intent>default</intent>
                </resourceObjects>
            </import>
        </work>
        <execution>
            <mode>none</mode> (1)
        </execution>
    </activity>
</task>
1 Avoids data processing.

The same effect can be attained by clicking on Reload button on the resource page for accounts, entitlements, or generics.

Impact on API Operations

The cached data are accessible by using the usual IDM Model Interface. There are two operation options that provide access to the cached data:

  • noFetch option: This option returns the data from midPoint repository. Therefore, if there are data cached in the repository then the noFetch option returns them.

  • staleness option: Requirement how stale or fresh the retrieved data should be. It specifies maximum age of the value in milliseconds. The default value is zero, which means that a fresh value must always be returned. This means that caches that do not guarantee fresh value cannot be used. If non-zero value is specified then such caches may be used. In case that Long.MAX_VALUE is specified then the caches are always used and fresh value is never retrieved.

Both options can be used to get cached data. The primary difference is that the noFetch option never goes to the resource, and it returns whatever data are in the repository. On the other hand, the staleness option is smarter, and it determines whether it has to go to the resource or not. In case that the "maximum" staleness option is used it will result in an error if cached data is not available.

Those options can be used both with getObject operations and search operations. For getObject the staleness option work as expected. But there is one special consideration for the search operations. The search operations cannot easily determine how fresh the data in the repository are. E.g. there may be new objects on the resource that are not in the repository. Therefore, to be on the safe side the search operations will always make search on the resource even if staleness option is specified. There is just one exception: the maximum staleness option will force repository search. However, if the search discovers any object that does not have cached data then it will result in an error (specified in the fetchResult object property).

Caching Metadata in Returned Shadows

Shadow Objects contain cachingMetadata property. This property can be used to determine whether the returned shadow represents fresh or cached data:

  • If no cachingMetadata property is present in the shadow then the data are fresh. They have been just retrieved from the resource.

  • If cachingMetadata property is present then the data are taken from the cache. The cachingMetadata property specified how fresh the data are (when they were originally retrieved).

Relation to the "Caching-Only" Read Capability

When the "caching only" read capability is present (e.g., for manual resources), the full shadow caching is enabled, with the following differences in default values (comparing to the standard caching):

Table 5. Default values differences for "caching-only" read capability
Item Standard default value Default when turned on by "caching-only" capability

timeToLive

P1D

unlimited

defaultCacheUse

cachedOrFresh

fresh

attributes caching scope

defined

all

credentials caching scope

all

none

The caching for "caching only" read capability can be turned off by specifying cachingStrategy to none.

The defaults above can be turned back to standard default values (the second column) by specifying cachingStrategy of passive.

Limitations

TODO describe these

Setting via Configuration Property

The default values for caching can be influenced by setting midpoint.internals.shadowCachingDefault configuration property, or midpoint/internals/shadowCachingDefault property in config.xml file.

If set to none, then the default behavior is to have caching turned off. If not set, or set to standard, the default is to have caching turned on.

Migration Note

Before 4.9, this feature was experimental. The default setting was that all attributes and no associations were cached.

Since 4.9, the defaults are more elaborate, as described in this document. Please take that into account when migrating.

See Also

Compliance

This feature is related to the following compliance frameworks:

Was this page helpful?
YES NO
Thanks for your feedback