Integrating Perforce Into Your Configuration

Deprecation Note

This tutorial was written with the tk-config-default configuration, not tk-config-default2, in mind. It refers to the original versions of the workfiles and publisher app. The newer apps, Workfiles 2 and Publisher 2, are not compatible with the hooks provided by the Perforce integration. Also, the structure of the configuration in tk-config-default2 has been heavily modified. As such, any file referenced by this tutorial is not going to match what you find in the newest configuration.

Foreword

For clarity and brevity, we're only going to be going through what is necessary for getting Perforce up and running using Maya and only for assets. Once you are familiar with how to set it up for one Software in one environment file, you'll be able to configure it for every Software and every environment file.

If you are not familiar with what our Perforce integration offers, we recommend you take a look at this page first.

LINKBOX_DOC:54:Perforce Integration

Overview

The Shotgun and Perforce integration is comprised of three major components.

1. The Shotgun server

2. The Perforce server

3. The Perforce to Shotgun synchronization daemon

The sync daemon will monitor commits as they are created on the Perforce server and will push the commit metadata back to the Shotgun server.

On the Toolkit side, you will need to alter the pipeline configuration so that a user can authenticate with Perforce and emit p4 commands to the server so that files are checked in and out appropriately.

Limitations

At the moment, the name of the user in Shotgun must match the name of the user in Perforce. This is an unfortunate limitation, but ensures that the daemon that synchronizes the Shotgun server with the data coming from Perforce knows who published new versions. We will revisit this limitation in the future.

Configuring Shotgun to track Perforce commits

From the Shotgun web app, click on the user icon, then pick Site Preferences. Expand the Entities section and enable the Revision entity. This will allow Shotgun to track Perforce commits.

Configuring Toolkit to connect with Perforce

In the following section, we will assume that you are already familiar at deploying Toolkit based pipelines. If you aren't, we recommend you read our guide on how to configure your first project.

LINKBOX_DOC:55:Beyond your first project

At the end of this section, you will be able to pick a workspace using the Perforce Connection app and run the Perforce to Shotgun synchronization daemon.

Common Perforce settings, apps and hooks

There are some boilerplate configurations that we will need in multiple environments, but we don't want to copy and paste them into multiple files. Instead, we'll put all those configurations in a single file, and then include that file into the various environment config files that use it. Create the file config/env/includes/perforce.yml in your pipeline configuration and paste in the following:

perforce-framework:
  hook_get_perforce_user: '{self}/get_perforce_user.py'
  hook_get_shotgun_user: '{self}/get_shotgun_user.py'
  hook_load_publish_data: default
  hook_load_review_data: default
  hook_store_publish_data: default
  hook_store_review_data: default
  host: ''
  location: {name: tk-framework-perforce, type: app_store, version: v0.1.13}
  server: tcp:localhost:1666
  server_aliases: []
  server_version: 2015.2
perforce_login:
  connect_on_startup: true
  hook_scene_operation: default
  location: {name: tk-multi-perforce, type: app_store, version: v0.1.2}

Now that we have all the common pieces centralized in a single file, we can start updating the configuration. We're going to assume at this point that you have a Perforce server up and running and are familiar with how to add files to the Perforce depot.

Configuring which server to use

Open config/env/includes/perforce.yml, find the server setting and set it to the address of your Perforce server. The default is tcp:localhost:1666.

Setting up your Perforce workspace

The Perforce workspace needs to contain all the files from the project root. For example, if your primary local file storage is Y:\shotgun and your project name is big_buck_bunny, then your workspace needs to at least include the entirety of the y:\shotgun\big_buck_bunny folder. Note that your Perforce workspace could actually be rooted at Y:\ and Y:\shotgun also.

Adding authentication with Perforce in Software

1. In config/env/project.yml, config/env/asset.yml and config/env/asset_step.yml, add the following to the includes section. This will ensure that our common yaml snippets can be used.

- ./includes/perforce.yml

2. In each of the files found in step 1, find the tk-maya section under engines and add the following under the apps section. This will ensure that you can authenticate with Perforce after Maya is launched regardless of the context.

tk-multi-perforce: '@perforce_login'

3. Find the frameworks section and add the following. This will make sure that the Perforce app can use the Perforce framework.

tk-framework-perforce_v0.x.x: '@perforce-framework'

4. Run ./tank cache_apps to download all the tools we've just added.

If you launch Maya at this point, you will be prompted to login into Perforce and pick a workspace. If you are not being prompted, please review the steps above.

Configuring the Perforce to Shotgun daemon

We need to setup a daemon that will synchronize your Shotgun server with the Perforce server. The way that the daemon operates is that it inspects the commits going into Perforce and then registers individual publishes into Shotgun. You need to find a computer that has access to the Shotgun server, the Perforce server and the pipeline configuration in order to run this daemon.

1. At the root of your Shotgun project, you will need to create the file tank/config/tank_configs.yml. Given the example above, that would be Y:\shotgun\big_buck_bunny\tank\config\tank_configs.yml.

2. In this file, we will put the path to the pipeline configuration for this project. For example, if your pipeline configuration was located at X:\shotgun\big_buck_bunny\pipeline, you will enter the following.

# Be careful to put double backslashes if using backslashes. Otherwise the yaml file reader will try to
# interpret "\s", "\b" and "\p" as single characters.
- win32: "X:\\shotgun\\big_buck_bunny\\pipeline" 
  linux2: "/mnt/shotgun/big_buck_bunny/pipeline"
  darwin: "/mnt/shotgun/big_buck_bunny/pipeline"

Note that traditionally in Toolkit, dictionaries such as this one are using the keys windows, linux and darwin. This is a bug that needs to be addressed on our end.

3. Commit this file into Perforce.

4. On the command line, type the following:

./tank install_app project tk-shell tk-shell-perforcesync

5. On the computer you have designated to run the daemon, launch the daemon using the following command:

./tank --debug sync_perforce_daemon -s 123456 -u <perforce user> -p <perforce password>

6. If you are using login based authentication to run the daemon, it is possible for your session to timeout if the daemon remains idle for too long. Therefore we recommend using script based authentication on the command-line to run the daemon. To learn more about this feature, type

./tank --help

on the command line and read the "General options and info" section at the beginning.

Note that --script-name and --script-key require that you upgrade the Toolkit Core to v0.17.24 or greater.

We recommend adding the --debug argument to know what is the current status of the daemon.

Note that the Perforce user needs to be able to update Perforce counters in order for the daemon to work correctly, so it's better to use a Perforce administrative account this the daemon.

If your Perforce server already has a lot of commits, we recommend starting it with -s and a commit id so that the sync daemon doesn't look at all commits on the server.

Configuring Toolkit apps

In the following section, we will assume you are starting a new project with no files currently in Perforce. We will first update your templates and file manager tool settings so that files are created without any version tokens, since Perforce will be tracking the versions. Then, we'll update the Publish app so we can start pushing files to Perforce. Finally, we'll update the Loader app settings so we can retrieve those files from Perforce.

Configuring the Shotgun File Manager app

At the moment, only the original Shotgun File Manager is supported with Perforce.

1. Inside config/core/templates.yml, find the following templates and update them so the version number is gone:

maya_asset_work: '@asset_work_area_maya/{name}.ma'
maya_asset_snapshot: '@asset_work_area_maya/snapshots/{name}.{timestamp}.ma'

2. Inside config/env/project.yml, in tk-maya's apps section, update the tk-multi-workfiles line to tk-multi-workfiles: '@workfiles-launch-at-startup-tk-maya'. This way we won't disrupt other app settings.

3. Do the same thing inside config/env/asset.yml.

4. In config/env/includes/common_apps.yml, copy the workfiles-launch-at-startup block and paste it below itself. Rename the pasted copy to workfiles-launch-at-startup-tk-maya and update the following settings:

workfiles-launch-at-startup-tk-maya:
  hook_filter_publishes: '{tk-framework-perforce_v0.x.x}/shared/filter_publishes.py'
  hook_filter_work_files: '{tk-framework-perforce_v0.x.x}/tk-multi-workfiles/filter_work_files.py'
  hook_scene_operation: '{tk-framework-perforce_v0.x.x}/tk-multi-workfiles/scene_operation_tk-maya.py'

5. In config/env/asset_step.yml, find the tk-multi-workfiles settings under the tk-maya section and update the following settings:

workfiles-launch-at-startup-tk-maya:
    hook_filter_publishes: '{tk-framework-perforce_v0.x.x}/shared/filter_publishes.py'
    hook_filter_work_files: '{tk-framework-perforce_v0.x.x}/tk-multi-workfiles/filter_work_files.py'
    hook_scene_operation: '{tk-framework-perforce_v0.x.x}/tk-multi-workfiles/scene_operation_tk-maya.py'
    template_publish: maya_asset_work
    template_publish_area: asset_work_area_maya
    template_work: maya_asset_work
    template_work_area: asset_work_area_maya

If you take a closer look at the template settings, you'll notice that both the work area and publish area now point to the same location, which means we will be publishing and working from the same location, in accordance to a Perforce workflow.

6. In the same file, update the includes and frameworks section just like we did in project.yml.

7. You can remove the settings for the tk-multi-snapshot app since it is not compatible with the Perforce workflow.

To test that everything works, simply create a new scene for a given task. There should be no version numbers in the file name.

Configuring the Publish app

1. Open config/env/asset_step.yml and find the tk-multi-publish settings for the tk-maya engine and update these settings:

tk-multi-publish:
  hook_post_publish: '{tk-framework-perforce_v0.x.x}/tk-multi-publish/post_publish.py'
  hook_primary_pre_publish: '{tk-framework-perforce_v0.x.x}/tk-multi-publish/primary_pre_publish.py'
  hook_primary_publish: '{tk-framework-perforce_v0.x.x}/tk-multi-publish/primary_publish.py'
  hook_scan_scene: '{tk-framework-perforce_v0.x.x}/tk-multi-publish/scan_scene_tk-maya.py'
  hook_secondary_pre_publish: '{tk-framework-perforce_v0.x.x}/tk-multi-publish/secondary_pre_publish_tk-maya.py'
  hook_secondary_publish: '{tk-framework-perforce_v0.x.x}/tk-multi-publish/secondary_publish_tk-maya.py'
  primary_publish_template: null
  secondary_outputs:
  - description: Submit all published files to Perforce
    display_group: Perforce
    display_name: Submit To Perforce
    icon: icons/p4_publish_submit.png
    name: p4_submit
    publish_template: null
    required: false
    scene_item_type: perforce_submit
    selected: true
    tank_type: ''

Note that in the secondary_outputs section of the tk-multi-publish settings, you can turn the selected setting to falseif you don't want Toolkit to commit files to Perforce on your behalf by default.

If you are using a Software other than Maya to test this, simply replace tk-maya in the previous hook settings with the Software you wish to use. You can find Perforce aware-versions of the publish hooks for 3dsmax, Photoshop and Maya here. If you are using another Software, you will need to roll out your own version of the hooks for the moment.

2. Inside config/core/templates.yml, you can optionally remove the lines for asset_publish_area_maya and maya_asset_publish as these won't be necessary anymore.

3. Copy this file to your config/icons folder in your pipeline configuration. This icon will be used in the publish tool's UI.

You are now ready to test publishing to Perforce.

1. Launch Maya.
2. Create a new scene in the file manager.
3. Add some geometry to it for testing.
4. Use the Shotgun File Manager to save the scene.
5. Use the Publish app to publish the scene.

If you switch over to the output of the Perforce sync tool, you should see something similar to this in the tool's output.

DEBUG [08:54:41 73.5399723053]: Looking for the next change submitted to 
Perforce...
DEBUG [08:54:41 75.3729343414]:  > Found change {'status': 'submitted',
'fileSize': ['74214'], 'changeType': 'public', 'rev': ['2'], 'client':
'jfboismenu_Jean-Francoiss-MacBook-Pro_1287', 'user': 'jfboismenu', 'time':
'1456148646', 'action': ['edit'], 'path':
'//depot/assets/Character/test/Art/work/maya/*', 'digest':
['2741D8E09CD05C6622E1C8494CBCE14A'], 'type': ['text'], 'depotFile':
['//depot/assets/Character/test/Art/work/maya/scene.v001.ma'], 'change': '4',
'desc': 'Shotgun publish\n'}
DEBUG [08:54:41 75.6020545959]: Checking that change '4' contains files that 
are in the current project... 
Creating Shotgun Revision entity for Perforce change 4
DEBUG [08:54:41 578.094005585]: Successfully created Shotgun Revision entity
14 for Perforce change 4
DEBUG [08:54:41 578.372955322]: Updating the Perforce counter
'tk_perforcesync_project_74' to 4

Of particular interest are the lines Creating Shotgun Revision entity for Perforce change 4 and Successfully created Shotgun Revision entity14 for Perforce change 4. This means the files were successfully synchronized with Shotgun.

Configuring the Loader app

1. In config/env/asset_step.yml, find the tk-multi-loader2 section and update the following settings:

tk-multi-loader2:
    filter_publishes_hook: '{tk-framework-perforce_v0.x.x}/shared/filter_publishes.py'
    actions_hook: '{tk-framework-perforce_v0.x.x}/tk-multi-loader2/tk-maya_actions.py'

That's it. Now you'll be able to import files into your scenes. To test,

1. Launch Maya.
2. Create a new scene in the file manager.
3. Use the Loader app to reference a file.

Note that if the file is not currently in your workspace, it will be copied from the server.