Troubleshooting SharePoint’s “Hidden List” and Managed Metadata Columns

Troubleshooting SharePoint’s “Hidden List” and Managed Metadata Columns

Posted by on Monday, September 23rd, 2013  

 

With SharePoint 2010 and 2013, lists and libraries support lookup columns which appear to retrieve data from the term stores of the Managed Metadata Service. However, using Managed Metadata Columns within a site collection creates a hidden list at the site collection level which contains a subset of term store information at <site collection URL>/Lists/TaxonomyHiddenList/AllItems.aspx.

While the content is obtained from the term store when in edit mode, the lookup target for views of the list (or library) is this hidden list at the root of the site collection which contains a local copy of the data. This is true whether the column is a native SharePoint Managed Metadata Column or a custom one created by autoclassification products such as Smartlogic Semaphore.

This hidden list is simply one instance of localized storage of copies of external data within a site collection. Accessing this local data enhances performance but sometimes creates issues with incorrect data being presented.

Understanding what is happening behind the scenes is essential in troubleshooting these issues. In this article, I will try to summarize what is happening. For more details, I suggest reading several articles which were useful in piecing together this article:

While most of these posts are based upon SharePoint 2010, little has changed with the hidden lists in SP2013.

Why a hidden list?

Local storage of data is easier and quicker for lookup columns to access for view presentations since direct access to the term store in the Managed Metadata Service would need to be through the published service URL.

A single, centralized list of all terms used by all MMS lookup columns in all lists or libraries within the site collection is smaller and simpler than replicating the values to each list and list item especially when values are changed in the term store.

What does the hidden list contain?

Essentially information on terms used by lists, their location within the MMS, and last updated values. Hidden list columns are:

Title: Name of the term

IdForTermStore: GUID of the term store or MMS instance

IdForTermSet: GUID of the term set.

IdForTerm:GUID of the term

Term:Term selected.

Path:Term path selected.

CatchAllData: Compressed GUID path used by search, target for TaxCatchAll column of list containing the lookup column

CatchAllDataLabel: Data used by search, target for TaxCatchAllLabel column of list containing the lookup column

Term[LCID]: Localized term for each language pack installed.

Path[LCID]: Localized term path for each language pack installed.
Note: If multiple language packs are installed and used in MMS, there will instances of the last two columns for each language pack.

Note: When a list is crawled by search, the values in the TaxCatchAll field will be used to create special crawled properties prefaced by ows_tax_Id mapped to a corresponding managed property prefaced by owstaxId.

Problems that I have encountered with the hidden list

The concept of the hidden list looks excellent. The implementation sometimes seems flaky to me. Here are some issues that I have encountered followed by some troubleshooting steps.

MMS (or Semaphore) columns on lists do not display information or users cannot edit the list.

This list has unique permissions.  Sometimes, the list is not created with the appropriate permissions. If your account has application level permissions, you may be able to create columns and edit content but users cannot.

The System account should have full control, and Authenticated Users should have Read permission. Correcting these settings for the hidden list should resolve access permissions.

Information in list views, the hidden list and search results do not agree

The information presented in views is retrieved first from a cache source then the hidden list. There is a timer job for each content web application named “Taxonomy Update Scheduler” which runs by default at some point each hour to update information on this list with the latest and greatest information from the term store.

Unfortunately, the system for determining that term store information is newer than that of the hidden list is not reliable. The term store maintains a simple change log. It isn’t really a true change log, it’s more a log of what terms are used on what site collections and a collection of Boolean values that indicate if the term has changed in the term store. You can see regular requests to the MMS for change information in ULS logs. The timer job checks that change log for items on the hidden list. Despite its name, the timer job does not update the taxonomy; it replaces information on the hidden list with changes in the term store IF the changes are detected and IF the pointer in the hidden list is correct. So, sometimes hidden lists are not updated because changes are not detected. And sometimes, only certain terms on the lists are not updated.

If the Taxonomy Update Scheduler timer job has been disabled or is not running successfully for some reason then old information is retained on all lists and libraries within the site collection using the hidden list information.

Since search updates the index from information in the hidden list, not the term store directly, the path and terms in search results will be erroneous until the hidden list is updated and crawled. I have seen conflicting information for the same term appear in different lists, different views, and search results.

Troubleshooting steps

First check the status and history of the Taxonomy Update Scheduler timer job. It must be running successfully for the hidden list to function correctly.

If the timer jobs and/or event receivers are corrupted, they are created by features named TaxonomyFieldAdded and TaxonomyTimerJobs. These features can be removed and recreated by deactivating and activating those features.

 

PowerShell cmdlets:

Uninstall-SPFeature “TaxonomyFieldAdded”

Install-SPFeature “TaxonomyFieldAdded”

Uninstall-SPFeature “TaxonomyTimerJobs”

Install-SPFeature “TaxonomyTimerJobs”

 

This PowerShell will force updates without using the timer job:

$siteUrl = “WebApplicationURL”

Get-SPWebApplication $siteUrl | Get-SPSite -limit ALL | foreach { [Microsoft.SharePoint.Taxonomy.TaxonomySession]::SyncHiddenList($_) }

This script is resource intensive as it updates all items on all hidden lists and should be run during off-hours for your farm.

I once found that a single manually applied tag was not updating in the hidden list which indicated that the pointer was incorrect or corrupted.

I took the following steps:

  1. Deleted the term from the hidden list.
  2. Went to known item that was manually tagged with the term and edited the properties.
  3. Removed the offending tag and saved properties.
  4. Reopened properties and manually added the term back as a tag.
  5. The hidden list was updated with the new term.

Unfortunately, steps 2 -4 would need to be taken on each list which used the term and locating them may be difficult. Editing the hidden list manually is not recommended and should be your last resort.

Note: Your search index will only be updated by at least an incremental crawl of content sources where the hidden lists have been updated.

 

Disclaimer
The sample scripts are not supported under any Summit 7 Systems standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Summit 7 Systems further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Summit 7 Systems, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Summit 7 Systems has been advised of the possibility of such damages.

Subscribe to RSS Feed

Sign Up for Newsletter

7

7

comments

    Feb 20
    2015

    JukkaK

    Hi!

    Great article!

    We have a farm that is migrated from 2010 -> 2013 on-prem.
    1. We change the name of the term in the termstore
    2. Taxonomy timer job runs successfully and hidden list is updated OK
    3. Old library items (pages library with metadata properties) are not updated with the new term. When these items are checked out the new term is updated immediately to the properties and the old cannot be chosen anymore.

    Any help? Which process is failing here? I have tried your script that force updates without using timer job but no success.

    Reply
      Jul 11
      2016

      Martin

      Hi Jukka

      We are experiencing the same problem as you do, have you found a solution?

      Thank, Martin

      Reply
        Jul 12
        2016

        Brian Laws

        Hi, Martin/Jukka. When the farm was upgrade, how was the term store migrated? Did you restore the MMS service application? It sounds to me like the terms are there but they don’t have the same GUID as before. It is the term GUID which is stored with the data, not the text of the term, so it is important that the taxonomy is migrated in such a way that the GUIDs are preserved.

        Also, in your MMS service application, check the Orphaned Terms node. By any chance, do you see the old terms in there?

        Reply
    Mar 22
    2016

    Shwetha

    Hi,

    We have a Semaphore Metadata column which is referring to the Taxonomy hidden list of a different site collection for some reason and hence when we try to edit the items , we get “List not found” error . Is there a way to link the column back to the existing site collection’s hidden list ?

    Reply
      Jul 12
      2016

      Brian Laws

      Hi, Shwetha. Have you resolved the issue yet? It sounds like quite a head-scratcher!

      Reply

Leave a Reply