Managing the content sources of a search collection

Search collections consist of one or more content sources. You can administer the content sources.

To work with content sources of a collection select Administration > Search Administration > Manage Search > Search Collections. Then select a search collection by clicking the collection name link. Portal Search displays the Content Sources panel. It shows the status of the selected search collection and lists its content sources and their status. It shows information related to the individual content sources, and lets you perform tasks on these content sources.

You can select the following option icons and perform the following tasks in relation to the search collection which you selected from the Search Collections list:

New Content Source. Use this option to create a new content source for the search collection that you selected from the Search Collections list. For detailed instructions refer to the portlet help.
Notes:
1. You can configure a search collection to cover multiple content source of different types. For example, you can combine portal sites, web sites, and local document collections.
2. The selectable options and data entry fields that are displayed under the different configuration tabs depend on which type of content source you select.
3. If you select Portal site, the appropriate data for your portal site is already complete.
4. If you select WCM site (Web Content Manager site), you need to fill in the appropriate data. For information about how to construct the URL for the content source refer to Seedlist 1.0 REST service API in the Web Content Manager documentation.
5. For some content sources you might need to enter sensitive data, such as a user ID and password. For example, this applies to secured WebSphere® Portal Express® sites or HTTP sites that require a user ID and password. To ensure encryption of this sensitive data when it is stored, update and run the file searchsecret.xml by using the XML configuration interface before creating the content source.
6. When you create a portal site content source in a portal cluster environment that is configured with SSL, you need to provide the cell security info for the web server and the nodes. For example, in a cluster with the cluster URL https://web_server/wps/portal, the primary node URL http://node_1:10039/wps/portal, and the secondary node URL http://node_2:10050/wps/portal you need to provide the user ID and password for the web server and both nodes 1 and 2.
7. Under the General parameters tab, you have to set the URL for the content source in a field Collect documents linked from this URL: . The crawler needs this URL for crawling. For information about how to construct the URL for the content source refer to Seedlist 1.0 REST service API in the Web Content Manager documentation.
  Note: A crawler failure can be caused by URL redirection problems. If this occurs, try by editing this field accordingly, for example, by changing the URL to the redirected URL.
8. For crawling a web site content source you can set a timeout under the General parameters tab under the option Stop collecting after (minutes): . This timeout works as follows:
  1. The timeout works only for web site content sources.
  2. The timeout works as an approximate time limit. It might be exceeded by some percentage.
  3. The crawl action is put in a queue. It might therefore take several minutes until it is executed and the time counter starts. It might therefore seem that the crawl takes longer than the timeout that you set.
  Therefore, when you start the crawl by clicking Start Crawler, allow for some time tolerance and be aware of the time required for crawls and imports and availability of documents.
9. Under the Advanced Parameter tab, the entry field for the Default Character Encoding contains the initial default value windows-1252, regardless of the setting for the Default Portal Language under Administration > Portal Settings > Global Settings. Enter the required default character encoding, depending on your portal language. Otherwise documents might be displayed incorrectly under Browse Documents.
10. Before you start the crawl, set the preferred language of the crawler user ID to match the language of the search collection that it crawls.
11. You start the initial crawl on a newly created content source by either of the following options:
  - After you created a new content source, click the Start Crawler icon. This starts an immediate crawl.
  - When you create the content source, define a schedule under the Schedulers tab. The crawl will start at the next possible time that you specified.
Refresh. Use this option to update the list of content sources and the status shown for this collection.
Select the following option icons and perform the following tasks on a content source:
- View Content Source Schedulers. Use this option to view and manage schedulers. This option is only available if you have defined schedulers for the content source.
- Start Crawler. Click this icon to start a crawl on the content source. This updates the contents of the content source by a new run of the crawler. While a crawl on the content source is running, the icon changes to Stop Crawler. Click this icon to stop the crawl. Portal Search refreshes different content sources as follows:
  - For website content sources, documents that were indexed before and still exist in the content source are updated. Documents that were indexed before, but no longer exist in the content source are retained in the search collection. Documents that are new in the content source are indexed and added to the collection.
  - For WebSphere Portal Express sites , the crawl adds all pages and portlets of the portal to the content source. It deletes portlets and static pages from the content source that were removed from the portal. The crawl works similarly to the option Regather documents from Content Source.
  - For IBM® Web Content Manager sites , Portal Search uses an incremental crawling method. Additionally to added and updated content, the Seedlist explicitly specifies deleted content. In contrast, clicking Regather documents from Content Source starts a full crawl; it does not continue from the last session, and it is therefore not incremental.
  - For content sources created with the Seedlist provider option, a crawl on a remote system that supports incremental crawling, such as IBM Connections, behaves like a crawl on a Web Content Manager site.
- Regather documents from Content Source. This deletes all existing documents in the content source from previous crawls and then starts a full crawl on the content source. Documents that were indexed before and still exist in the content source are updated. Documents that were indexed before, but no longer exist in the content source are removed from the collection. Documents that are new in the content source are indexed and added to the collection.
- Verify Address of Content Source. Click this icon to verify that the URL of the content source is still live and available. Manage Search returns a message about the status of the content source.
- Edit Content Source. Click this icon to make changes to a content source. This includes configuring parameters, schedules, and filters for the selected content source.
  Notes:
  - It is of benefit to define a dedicated crawler user ID. The pre-configured default portal site search uses the default administrator user ID wpsadmin with the default password of that user ID for the crawler. If you changed the default administrator user ID during your portal installation, the crawler uses that default user ID. If you have made changes to the user ID or password for the administrative user ID and still want to use that user ID for the Portal Search crawler, you need to adapt the settings here accordingly.
    To define a crawler user ID, select the Security tab, and update the user ID and password. Click Save to save your updates.
  - If you modify a content source that belongs to a search scope, update the scope manually to make sure that the scope still covers that content source. Especially if you changed the name of the content source, edit the scope and make sure that it is still listed there. If not, add it again.
- Delete Content Source. Click this icon to delete the selected content source.
  Note: If you delete a content source, then the documents that were collected from this content source will remain available for search by users under all scopes which included the content source before it was deleted. These documents will be available until their expiration time ends. You can specify this expiration time under Links expire after (days): under General Parameters when you create the content source.
View information about the status and configuration of the content source.
Note: To update the status information, click the Refresh button or the refresh button of the browser.
View the Collection Status information of the selected search collection. The status fields show the following data that changes over the lifetime of the search collection:
Search Collection Name:

Shows the name of the selected search collection.

Search Collection Location:

Shows the location of the selected search collection in the file system. This is the full path where all data and related information of the search collection is stored.

Collection Description:

Shows the description of the selected search collection if available.

Search Collection Language:

Shows the language for which the search collection and its index are optimized. The index uses this language to analyze the documents when indexing, if no other language is specified for the document. This feature enhances the quality of search results for users, as it allows them to use spelling variants, including plurals and inflections, for the search keyword.

Summarizer used:

Shows whether a static summarizer is enabled for this search collection.

Last update completed:

Shows the date when a content source defined for the search collection was last updated by a scheduled update.

Next update scheduled:

Shows the date when the next update of a content source defined for the search collection is scheduled.

Number of active documents:

Shows the number of active documents in the search collection, that is, all documents that are available for search by users.
Notes:
1. To update the status information, click Refresh. Clicking the refresh button of the browser will not update the status information.
2. If you delete a portlet from the portal after a crawl of the portal site, the deleted portlet is no longer listed in the search results. However, refreshing the view does not update the status information about the Number of active documents. This information is not updated until after the next cleanup run of portal resources.

For more details about the available options for content sources, refer to the Manage Search portlet help.