Skip to main content

Migrating Drupal 7 File Entities to Drupal 8 Media Entities

The Drupal 8.3.x branch is getting ready to introduce a new experimental media module. This will bring enhanced media handling in Drupal 8. The closest solution in Drupal 7 to handle media is the file entity module. Now is the time to discuss migrations from file entity in Drupal 7 to media entities in Drupal 8. For core, there is already an issue for this, but for contrib... there is no migration. So, I wrote one.

by jibran.ijaz /

The file_entity module adds a type field to an existing {file_managed} table which allows creating file bundles for each file type. There is already a Drupal 7 to Drupal 8 files migration in core. Managed files are not fieldable entities in Drupal 7 so the file migration in core doesn't handle the file fields but file_entity module allows to add fields to file bundles.

To add support for 'types' and make the migration fieldable I wrote a new migrate source plugin. It adds a type filter to the query to allow migrating specific files. Also, I made the plugin extend from the FieldableEntity source plugin in core. To import the fields for the file type, I override the prepareRow method in the plugin.

The final source plugin looks like this:


<?php
// modules/custom/my_custom_module/src/Plugin/migrate/source/FileEntity.php
namespace Drupal\my_custom_module\Plugin\migrate\source;

use Drupal\Core\Database\Query\Condition;
use Drupal\migrate\Row;
use Drupal\migrate_drupal\Plugin\migrate\source\d7\FieldableEntity;

/**
 * Drupal 7 file_entity source from database.
 *
 * @MigrateSource(
 *   id = "file_entity",
 *   source_provider = "file"
 * )
 */
class FileEntity extends FieldableEntity {

  /**
   * {@inheritdoc}
   */
  public function query() {
    $query = $this->select('file_managed', 'f')
      ->fields('f')
      ->orderBy('f.fid');
    if (isset($this->configuration['type'])) {
      $query->condition('f.type', $this->configuration['type']);
    }
    // Filter by scheme(s), if configured.
    if (isset($this->configuration['scheme'])) {
      $schemes = array();
      // Accept either a single scheme, or a list.
      foreach ((array) $this->configuration['scheme'] as $scheme) {
        $schemes[] = rtrim($scheme) . '://';
      }
      $schemes = array_map([$this->getDatabase(), 'escapeLike'], $schemes);

      // The uri LIKE 'public://%' OR uri LIKE 'private://%'.
      $conditions = new Condition('OR');
      foreach ($schemes as $scheme) {
        $conditions->condition('uri', $scheme . '%', 'LIKE');
      }
      $query->condition($conditions);
    }

    return $query;
  }

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {
    // Get Field API field values.
    foreach (array_keys($this->getFields('file', $row->getSourceProperty('type'))) as $field) {
      $fid = $row->getSourceProperty('fid');
      $row->setSourceProperty($field, $this->getFieldValues('file', $field, $fid));
    }
    return parent::prepareRow($row);
  }

  /**
   * {@inheritdoc}
   */
  public function fields() {
    return array(
      'fid' => $this->t('File ID'),
      'uid' => $this->t('The {users}.uid who added the file. If set to 0, this file was added by an anonymous user.'),
      'filename' => $this->t('File name'),
      'uri' => $this->t('The URI to access the file'),
      'filemime' => $this->t('File MIME Type'),
      'status' => $this->t('The published status of a file.'),
      'timestamp' => $this->t('The time that the file was added.'),
      'type' => $this->t('The type of this file.'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    $ids['fid']['type'] = 'integer';
    return $ids;
  }

}

There were four file types in Drupal 7.

  • Audio
  • Image (local)
  • Videos (local)
  • Videos (youtube)

I installed four modules for this.

I also created four media bundles for these media type plugins.

Before writing the actual migrations, the file migration was needed.


# modules/custom/my_custom_module/migrations/my_files.yml
id: my_files
label: Files
migration_tags:
  - Custom
source:
  plugin: d7_file
  constants:
    source_base_path: 'sites/default/files/'
    old_files_path: 'sites/default/files/migration-files'
process:
  filename: filename
  source_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/old_files_path
        - filepath
    -
      plugin: urlencode
  uri:
    -
      plugin: skip_youtube_files
      source:
        - '@source_full_path'
        - uri
    -
      plugin: file_copy
  filemime: filemime
  # filesize is dynamically computed when file entities are saved, so there is
  # no point in migrating it.
  # filesize: filesize
  status: status
  created: timestamp
  changed: timestamp
  fid: fid

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

destination:
  plugin: entity:file
  migration_dependencies:
    required:
      - my_users

In Drupal 7, the media_youtube module is using Youtube stream wrapper to store the files. To ignore Youtube videos, we wrote a process plugin.


<?php
# modules/custom/my_custom_module/src/Plugin/migrate/process/SkipYoutubeVideos.php
namespace Drupal\my_custom_module\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\MigrateSkipRowException;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;

/**
 * Skip youtube videos.
 *
 * @MigrateProcessPlugin(
 *   id = "skip_youtube_files"
 * )
 */
class SkipYoutubeVideos extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    if (parse_url(end($value), PHP_URL_SCHEME) == 'youtube') {
      throw new MigrateSkipRowException();
    }
    return $value;
  }

}

We kept the file id same between Drupal 7 and Drupal 8 so I decided to keep media entity id same as file id as well for the media migration. Each file type had a different set of fields so I wrote per bundle migrations.

To import audio files I configured the file type in migration also hard coded the target bundle to audio.


# modules/custom/my_custom_module/migrations/my_media_audio.yml
id: my_media_audio
label: Media Audio
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: audio
  constants:
    bundle: 'audio'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # File field see media_entity.bundle.audio.yml.
  field_media_audio/target_id: fid
  # Title field.
  field_title: field_title
  # Transcript field.
  field_transcript: field_transcript

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

To migrate media images, I moved the dedicated field entity fields for alt and title to the image field.


# modules/custom/my_custom_module/migrations/my_media_audio.yml
id: my_media_image
label: Files
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: image
  constants:
    bundle: 'image'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # Image field see media_entity.bundle.image.yml.
  field_media_image/target_id: fid
  field_media_image/alt: field_file_image_alt_text/0/value
  field_media_image/title: field_file_image_title_text/0/value

  # Description field.
  field_description: field_image_description
  # Caption field.
  field_caption: field_caption

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

To import local video, I add the URI scheme to the video migration.


# modules/custom/my_custom_module/migrations/my_media_local_video.yml
id: my_media_local_video
label: Files
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: video
  # See output of SELECT DISTINCT(SUBSTRING_INDEX(uri, ':', 1))  FROM file_managed WHERE type = 'video';
  scheme:
    - "public"
  constants:
    bundle: 'local_video'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # File field see media_entity.bundle.local_video.yml.
  field_media_video/target_id: fid
  # Title field.
  field_title: field_video_title
  # Transcript field.
  field_transcript: field_transcript

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

To import the youtube videos the URI was like youtube://v/video_id but embed field needs the youtube video URL so I created a process plugin to convert the video ID to URL.


# modules/custom/my_custom_module/migrations/my_media_video.yml
id: my_media_video
label: Files
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: video
  # See output of SELECT DISTINCT(SUBSTRING_INDEX(uri, ':', 1))  FROM file_managed WHERE type = 'video';
  scheme:
    - "youtube"
  constants:
    bundle: 'video'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # Embed field see media_entity.bundle.video.yml.
  field_media_video_embed_field:
    plugin: youtube
    source: uri
  # Title field.
  field_title: field_video_title
  # Transcript field.
  field_transcript: field_transcript

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

The Youtube process plugin


<?php
# modules/custom/my_custom_module/src/Plugin/migrate/process/Youtube.php
namespace Drupal\my_module\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;

/**
 * Custom process plugin to convert youtube scheme uri to video url.
 *
 * @MigrateProcessPlugin(
 *   id = "youtube"
 * )
 */
class Youtube extends ProcessPluginBase {

  const SCHEME = 'youtube://';
  const BASE_URL = 'http://youtube.com/watch?';

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    // Convert youtube scheme uri to video url.
    if (strpos($value, static::SCHEME) !== FALSE) {
      $value = static::BASE_URL . implode('=', explode('/', str_replace(static::SCHEME, '', $value), 2));
    }
    else {
      $value = NULL;
    }
    return $value;
  }

}

I ran these migrations using migrate_drush.

Conclusion

On paper, file entity to media migration sounds difficult, but the migration API in Drupal core and its use of new entity and plugin APIs made the migration for a custom entity with complex fields a lot easier.

Thanks, to Lee Rowlands for writing the skip_youtube_files plugin. Also, thank you, Ben Dougherty, for all the advice.

The code can also be found as a gist.