Skip to main content
Skip table of contents

Getting Started with Dynamic Storage Utilization (DSU)

Getting started image

Purpose of this Guide

The purpose of this guide is to familiarize you with a feature being introduced in PeerGFS v5.0: Dynamic Storage Utilization (DSU).  It will introduce you to the feature set, UI, and workflows of DSU and step you through setting up and running a job that uses DSU.

This guide assumes general familiarity with using PeerGFS.  You should be comfortable with creating and running file collaboration and/or file synchronization jobs.  This guide focuses on DSU, not on the broader capabilities of PeerGFS.  See the PeerGFS User Guide for your version of PeerGFS for more information about other features and settings.

If you are not familiar with creating and running file collaboration and/or file synchronization jobs, please discuss this with your Peer Software point-of-contact.

Introduction to Dynamic Storage Utilization

Starting in PeerGFS v5.0, our new Dynamic Storage Utilization (DSU) feature allows you to save storage space on edge storage devices (for example, storage devices used in branch offices) where only a small subset of files are used on a frequent basis.  Files that are used less frequently are replaced with stub files on the edge storage device so that it appears to have a complete set of files.  When a user accesses a stub file, DSU retrieves the full version of the file from a master storage device.  The benefit of using DSU is that it allows you to efficiently utilize storage capacity on edge devices while preserving fast access performance on files that are used most often.

DSU offers flexible edge storage management with:

  • The ability to assign an amount or percentage of available storage to be used on the edge storage device.
  • Dynamic adjustments of the time periods used to determine whether to stub or rehydrate a file, allowing DSU to keep the assigned storage space as full as possible (best experience for the end user).
  • Direct integration with our file collaboration and synchronization job types.
  • Point-to-point data transfer capability between one edge and one or more masters.
  • The flexibility to mix and match master and edge roles across different jobs.
  • The ability to pin files or folders to always be local or always be stubbed on the edge storage device.
  • Alerting, to ensure you stay ahead of potential storage capacity limits.

Important Concepts

A master participant has a complete set of hydrated files and no stub files.  An edge participant contains a subset of the complete, hydrated files on a master participant, while the rest of the files are stub files that don't take up any space.  Users can retrieve stubbed files directly from a master participant as needed.  The goal of Dynamic Storage Utilization (DSU) is to keep as much as possible cached locally on edge participants for rapid access.

Every edge participant must have at least one master participant assigned to it.  When a stub file needs to be rehydrated, DSU will retrieve the file from a master participant.

User-defined business rules (volume and utilization policies) manage the storage capacity on edge devices. DSU scans edge participants on a set basis (typically at least once daily) and uses these policies to determine whether adjustments are needed, i.e., whether to stub files to free up space or to rehydrate files.  This ensures that the storage capacity is being used at optimum efficiency.

Glossary of DSU Terms

This glossary is provided to help you understand some of the core terminology used in conjunction with DSU.

Edge participant

A subset of the files stored on a master participant are physically stored on an edge participant; the rest of the files on an edge participant are stub files that take up minimal space but can be rehydrated as needed.

Local file

A file that is fully available without network access to a master participant; all of its bytes are present (stored locally) on the participant.

Master Data Service

A service that handles requests from edge participants for files on a master participant.  The Master Data Service is installed on the Peer Agent server as part of the Peer Agent installation process.

Master participant

Always has a complete set of files for the job. None of its files are stubbed; they are always stored physically on that device.

Pinning filter

Specifies whether specific files or files in a particular directory are always stubbed or always local on the edge participant.  A pinning filter similar is to a utilization policy—it can be applied to multiple jobs.  If there is a conflict between a pinning filter and utilization policy (where, for example, you might have something set to be always stubbed), the pinning filter will take precedence.

Rehydrated file

A file that was stubbed but has been fully reconstituted on the edge participant.

Stub file

A file that appears to the user to be stored on the local disk and immediately available for use but is actually held either in part or entirely on a different storage medium.

Temporary storage space

Space that is used to temporarily store the content of stub files that are being rehydrated.

Utilization policy

Defines when a file should be stubbed versus fully hydrated across all volumes of this edge participant.  Parameters are based on the size of the files to be potentially stubbed and when they were last accessed and modified.  A utilization policy enables you to balance getting the best performance while keeping the cache as full as possible.

Volume policy

Specifies how much of the available space on the volume monitored by the Agent/edge participant to be assigned for local (hydrated) files.

Requirements

  • The requirements for PeerGFS (including both DSU and enterprise NAS support) can be found here.
  • It is important to note that all our support enterprise NAS platforms can be used as master participants but only Windows File Servers can be used as edge participants.
  • We strongly recommend having at least two master participants for every job that uses DSU.  If one of the master participants goes down, another master participant can take over and serve data to the edges.

Create a Job that Uses DSU

The process for creating a job remains essentially the same; the primary change occurs when creating participants, where you must designate a participant as a master or edge participant.  See the following sections for instructions on how to create a job that utilizes DSU.

Create and name the job

  1. Open Peer Management Center.
  2. From the File menu, select New Job.
  3. Click File Collaboration or File Synchronization, and then click Create.
    Create New Job dialog
  4. Enter a name for the job and click OK.
    Create File Collaboration Job dialog

Create the job participants

You are now ready to create the job participants. A participant can be designated as either a master or edge participant.

To effectively deploy DSU, we recommend that the job has at least four participants. At least two of those should be master participants, so that if one master participant goes offline, the other master participant can continue to serve as source for files for the edge participant.

Create a master participant

  1. Click the Add button.
    Participants page - Create File Collaboration Job
  2. Select an Agent, and then click Next.
    Management Agent page - Add New Participant
  3. Select a storage platform, and then click Next.
    A master participant can be any type of storage device.


  4. If you selected Windows File Server, click Next on the Storage Information page. If you selected any other type of storage platform, select existing credentials or enter new credentials.
  5. Click Next.
  6. On the Path page, enter the path to the watch set.
    Path page - Add New Participant
  7. Select Seeding Target if you want this participant to be a seeding target, and then click OK.
    You must have at least one master participant that is not a seeding target.
  8. Click Next.
  9. Select the Enable Dynamic Storage Utilization checkbox.
    Dynamic Storage Utilization page - Add New Participant

    The Master role is selected by default.
    Dynamic Storage Utilization page - Add New Participant
  10. Click Next.
    The Master Data Service page appears. The Master Data Service handles requests for rehydrating stubbed files.   If the Agent you selected is already being used as a master participant in another job utilizing DSU, then the existing parameters for the Master Data Service will be displayed. You can edit the values by clicking Edit Maser Data Service. Any modifications you make will be applied to every other job that uses this Agent as a master participant.
    Master Data Service page - Add New Participant
  11. (Optional) Enter a value in the Agent Alias field.
    A value is required only if the name of the Agent server is different from the Agent name (for example, if the master participant's Windows server is named "Server1.Lab.local" but the edge participant can't resolve "Server1.Lab.local" to an IP because the edge is not on the domain at all).
    The value for Agent Alias can be the hostname, FDQN, or IP address of the server on which the Agent is installed.  If a value is entered, it will be used by the edge service on edge participants to connect with this master data service.  If no value is entered, the name of the Agent server will be used.
  12. (Optional) If necessary, modify the port number of the server that the Agent is installed on.
  13. Click Finish.
    The Participants page appears; the participant has been added to the Participants table.
    Participants page - Create File Collaboration Job
  14. Repeat the steps in this section to add additional master participants.

Create an edge participant

  1. Click the Add button.
  2. Select an Agent, and then click Next.
  3. Select Windows Files Server as the storage platform, and then click Next.
    For an edge participant, the storage platform must be Windows Files Server.
  4. Click Next on the Storage Information page.
  5. Click Next.
  6. On the Path page, enter the path to the watch set.
  7. Select Seeding Target if you want this participant to be a seeding target, and then click OK.
  8. Select the Enable Dynamic Storage Utilization checkbox, and under Dynamic Storage Utilization Role, select Edge.
  9. Click Next.
Create a volume policy
  1. Enter the root drive of the path specified on the Path page (e.g., C:) in the Temporary Storage Path field.
    Volume Policy page - Dynamic Storage Utilization

    A subfolder named .PeerTempPath will be created under the location that you specify and DSU will store temporary file blocks in that folder.
    Cache Size - Dynamic Storage Utilization
  2. Under Cache Size, we recommend keeping the default percentage of 75%.
    The percentage represents the maximum amount of disk space you want to allocate to DSU for fully hydrated files on the volume specified by the path on the Path page.
  3. For Cache Utilization, we recommend keeping the default value of 80%.
  4. For Cache Scan Schedule, accept the default value of a scan every day at 10 p.m., or set your preferred nightly schedule.
  5. Click Next.
Create a utilization policy
  1. Enter a name for the policy.
    Utilization Policy -  Dynamic Storage Utilization
  2. Leave the remaining options at their default values.  If you want to create a pinning filter, see the appropriate PeerGFS User Guide for instructions.
  3. Click Finish.
    The Participants page appears; the participant has been added to the Participants table.
  4. Repeat the steps in the Create an edge participant section to add additional edge participants.

Assign master participants to edge participants

For each edge participant, assign at least one master participant to it and set the failover order.  DSU will use the master participants when reading or rehydrating a stub file on the edge participant.

  1. Select an edge participant in the Master Configuration table, and then click Assign.
    Assign Master Participant -  Dynamic Storage Utilization
  2. In the Select Master Participants dialog, set the failover order by selecting a master participant and then using the Move Up and Move Down buttons to order the participants.
    Failover Order -  Dynamic Storage Utilization
  3. Click OK.
  4. Repeat for each edge participant.
  5. Click Next.

Create DSU email alerts

After creating a DSU-enabled job, we recommend creating DSU email alerts that flag you to potential or actual problems that allow you stay ahead of potential storage capacity limits. For example, cache utilization specifies the maximum percentage of the cache to be allocated to fully hydrated files.  When the maximum percentage is reached, you will receive an alert.
To create a DSU email alert:

  1. Select Preferences from the Window menu.
  2. Expand Collab, Sync, and Replication in the navigation tree, and then select Dynamic Storage Utilization.
  3. Select Email Alerts in the navigation tree, and then click Create.
  4. Enter a name for the alert.
  5. Create DSU Email Alert.
    DSU Email Alert
  6. Select the caching scan events to be alerted.
  7. Select the volume event types to be alerted
  8. Select Master/Edge Services Health Monitoring if you want to be alerted if either the Peer Master Data Service or the Peer Edge Service goes down.
  9. Enter alert recipients, and then click Add to List.
  10. Click OK to close the dialog.
  11. The new email alert can now be applied to DSU-enabled jobs.
  12. Click Apply and Close or Apply.
  13. After creating the alert, you will need to edit the job to apply the alert to the job.  Specifically, you will need to edit the volume policy for a participant.

Edit the job to apply DSU email alerts

If you have created DSU email alerts, you can now apply those alerts to a job utilizing DSU.

  1. In the Jobs view, right-click the job for which you want to apply alerts and select Edit Job.
    The Edit Job wizard opens.
    Edit File Collaboration Job wizard
  2. Select a participant in the Participants table.
  3. Click the Edit button.
    The Edit Participant page appears.
    General page - Edit Participant
  4. Click Next.
    The Dynamic Storage Utilization page appears.
    Dynamic Storage Utilization page - Edit Participant
  5. Click Next.
    The Volume Policy page appears.
    Volume Policy page - Edit Participant
  6. On the Volume Policy page, click the Edit Volume Policy link.
    The Volume Policies page appears.
    Volume Policies table - page - Edit Participant
  7. In the Volume Policies table, select the participant that you selected in Step 2, and then click the Edit button.
  8. Click OK in the In Use Configuration dialog that appears.
    In Use Configuration message - page - Edit Participant

    The Edit Volume Policy page appears.
    Edit Volume Policy page - page - Edit Participant
  9. Click Next.
    The Email Alerts page appears.  The table on this page lists the email alerts already applied.
    Email Alerts page - Edit Participant
  10. Click the Select button.
    The Select Email Alert page appears.
  11. Select an alert from the Email Alert dropdown, and then click OK.
    The selected alert appears in the table.
    Email Alerts page - Edit Participant
  12. Add additional alerts if desired.
  13. Click Finish.
  14. Click OK in the Data Change dialog.
    Data Change dialog - Edit Participant
  15. Click Apply and Close in the Volume Policies page.
    Volume Policies page - Preferences
  16. Click Finish in the Volume Policy page.
    Volume Policy summary - Edit Participant
  17. Click OK to close the Edit Job wizard.

Get Feedback on Your DSU Jobs

Determine the State of a File on an Edge Participant

You can determine whether a file is a stub on the edge participant in one of two ways:

Method 1

Look for the offline X in the bottom left corner of the file icon, (as seen on the the left-hand file in the below example).  The offline X should disappear.

X denoting offline file on edge participant

Method 2

Display the Attributes column in Windows Explorer and look for the L and O attributes.
If you open that file on a client system that is using SMB to access the edge participant, it will be automatically rehydrated with file data pulled from one of the master participants.  The offline X should disappear, as should the L and O attribute attributes.

L and O attributes

L and O attributes in Windows Explore

No L and O attributes

No L and O attributes in Windows Explorer

View DSU Job Statistics

You can get feedback about how DSU is performing in your environment. To view DSU job statistics, you must start DSU-enabled jobs. When a DSU job starts:

  • Connectivity to all Agents and other file storage devices is checked.
  • The real-time monitoring engine is initialized.
  • A background scan is kicked off to ensure all file servers are in sync with another.
  • If an edge participant is new to a job, the initial data will be replicated to that participant as stub files.
  • Once the initial background scan is complete, a cache optimization scan will be run on each edge participant.
  1. In the Jobs view, right-click one or more of your DSU jobs, and then select Start.

    Jobs view

  2. Double-click on a specific job in the Jobs view to see its runtime information and statistics.
  3. Double-click on the File Collaboration or File Synchronization job type in the Jobs view, and then click the Dynamic Storage Utilization tab to see details about how DSU is performing in your environment.  Note that it may take some time for statistics to be displayed after starting a job for the first time.

Examine Application-Specific Interaction

We recommend that you test DSU's interaction with your other applications.  For example, some applications may not like having their files stubbed or may not like the additional time it takes from the time the application tries to open a stub file to when DSU is able to get full file data from a master (an application may be used to opening a 1 GB file locally within a few seconds). 

To discover any potential compatibility issues between DSU and your line of business applications, we suggest that you:

  1. Make a list of the key applications that you and your end users use in production.
  2. For each available application, perform a series of normal operations (create, open, modify, etc.) against new and legacy files:
    • All operations performed against new files should be replicated to and fully available on all participants.
    • Opens of older files may take longer on the edge participants as the Edge service receives file data from the Master Data service over the network.

All testing should be performed against both master and edge participants from a client system that is not hosting a Peer Agent.  Client systems can be running client versions of Windows or the PMC server can be used as a client system.

Test Stubbing and Rehydration

Purpose

We recommend that in addition to testing application compatibility with DSU, you also test stubbing and rehydration to ensure that your network can handle the additional traffic required to rehydrate files from a master.

Data to Test

Testing of the stubbing and rehydration functionality is best done with older data (1 year or older).  That said, if the amount of local data on an edge participant does not fill its assigned cache size, DSU will dynamically rehydrate older files to ensure a better end user experience.

Test Process

By default, DSU will perform a nightly cache optimization scan of all files on each edge participant to determine if they should be stubbed or rehydrated.  To force the cache scan to run, right-click on a volume in the Dynamic Storage Utilization tab of the Collab, Sync, and Repl Summary view and click Run Cache Scan.

Run Cache Scan to test DSU


JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.