Skip to main content

Migrating Drupal 7 File Entities to Drupal 8 Media Entities

The Drupal 8.3.x branch is getting ready to introduce a new experimental media module. This will bring enhanced media handling in Drupal 8. The closest solution in Drupal 7 to handle media is the file entity module. Now is the time to discuss migrations from file entity in Drupal 7 to media entities in Drupal 8. For core, there is already an issue for this, but for contrib... there is no migration. So, I wrote one.

by Jibran Ijaz /

The file_entity module adds a type field to an existing {file_managed} table which allows creating file bundles for each file type. There is already a Drupal 7 to Drupal 8 files migration in core. Managed files are not fieldable entities in Drupal 7 so the file migration in core doesn't handle the file fields but file_entity module allows to add fields to file bundles.

To add support for 'types' and make the migration fieldable I wrote a new migrate source plugin. It adds a type filter to the query to allow migrating specific files. Also, I made the plugin extend from the FieldableEntity source plugin in core. To import the fields for the file type, I override the prepareRow method in the plugin.

The final source plugin looks like this:

 
<?php
// modules/custom/my_custom_module/src/Plugin/migrate/source/FileEntity.php
namespace Drupal\my_custom_module\Plugin\migrate\source;

use Drupal\Core\Database\Query\Condition;
use Drupal\migrate\Row;
use Drupal\migrate_drupal\Plugin\migrate\source\d7\FieldableEntity;

/**
 * Drupal 7 file_entity source from database.
 *
 * @MigrateSource(
 *   id = "file_entity",
 *   source_provider = "file"
 * )
 */
class FileEntity extends FieldableEntity {

  /**
   * {@inheritdoc}
   */
  public function query() {
    $query = $this->select('file_managed', 'f')
      ->fields('f')
      ->orderBy('f.fid');
    if (isset($this->configuration['type'])) {
      $query->condition('f.type', $this->configuration['type']);
    }
    // Filter by scheme(s), if configured.
    if (isset($this->configuration['scheme'])) {
      $schemes = array();
      // Accept either a single scheme, or a list.
      foreach ((array) $this->configuration['scheme'] as $scheme) {
        $schemes[] = rtrim($scheme) . '://';
      }
      $schemes = array_map([$this->getDatabase(), 'escapeLike'], $schemes);

      // The uri LIKE 'public://%' OR uri LIKE 'private://%'.
      $conditions = new Condition('OR');
      foreach ($schemes as $scheme) {
        $conditions->condition('uri', $scheme . '%', 'LIKE');
      }
      $query->condition($conditions);
    }

    return $query;
  }

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {
    // Get Field API field values.
    foreach (array_keys($this->getFields('file', $row->getSourceProperty('type'))) as $field) {
      $fid = $row->getSourceProperty('fid');
      $row->setSourceProperty($field, $this->getFieldValues('file', $field, $fid));
    }
    return parent::prepareRow($row);
  }

  /**
   * {@inheritdoc}
   */
  public function fields() {
    return array(
      'fid' => $this->t('File ID'),
      'uid' => $this->t('The {users}.uid who added the file. If set to 0, this file was added by an anonymous user.'),
      'filename' => $this->t('File name'),
      'uri' => $this->t('The URI to access the file'),
      'filemime' => $this->t('File MIME Type'),
      'status' => $this->t('The published status of a file.'),
      'timestamp' => $this->t('The time that the file was added.'),
      'type' => $this->t('The type of this file.'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    $ids['fid']['type'] = 'integer';
    return $ids;
  }

}

There were four file types in Drupal 7.

  • Audio
  • Image (local)
  • Videos (local)
  • Videos (youtube)

I installed four modules for this.

I also created four media bundles for these media type plugins.

Before writing the actual migrations, the file migration was needed.


# modules/custom/my_custom_module/migrations/my_files.yml
id: my_files
label: Files
migration_tags:
  - Custom
source:
  plugin: d7_file
  constants:
    source_base_path: 'sites/default/files/'
    old_files_path: 'sites/default/files/migration-files'
process:
  filename: filename
  source_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/old_files_path
        - filepath
    -
      plugin: urlencode
  uri:
    -
      plugin: skip_youtube_files
      source:
        - '@source_full_path'
        - uri
    -
      plugin: file_copy
  filemime: filemime
  # filesize is dynamically computed when file entities are saved, so there is
  # no point in migrating it.
  # filesize: filesize
  status: status
  created: timestamp
  changed: timestamp
  fid: fid

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

destination:
  plugin: entity:file
  migration_dependencies:
    required:
      - my_users

In Drupal 7, the media_youtube module is using Youtube stream wrapper to store the files. To ignore Youtube videos, we wrote a process plugin.


<?php
# modules/custom/my_custom_module/src/Plugin/migrate/process/SkipYoutubeVideos.php
namespace Drupal\my_custom_module\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\MigrateSkipRowException;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;

/**
 * Skip youtube videos.
 *
 * @MigrateProcessPlugin(
 *   id = "skip_youtube_files"
 * )
 */
class SkipYoutubeVideos extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    if (parse_url(end($value), PHP_URL_SCHEME) == 'youtube') {
      throw new MigrateSkipRowException();
    }
    return $value;
  }

}

We kept the file id same between Drupal 7 and Drupal 8 so I decided to keep media entity id same as file id as well for the media migration. Each file type had a different set of fields so I wrote per bundle migrations.

To import audio files I configured the file type in migration also hard coded the target bundle to audio.


# modules/custom/my_custom_module/migrations/my_media_audio.yml
id: my_media_audio
label: Media Audio
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: audio
  constants:
    bundle: 'audio'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # File field see media_entity.bundle.audio.yml.
  field_media_audio/target_id: fid
  # Title field.
  field_title: field_title
  # Transcript field.
  field_transcript: field_transcript

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

To migrate media images, I moved the dedicated field entity fields for alt and title to the image field.


# modules/custom/my_custom_module/migrations/my_media_audio.yml
id: my_media_image
label: Files
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: image
  constants:
    bundle: 'image'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # Image field see media_entity.bundle.image.yml.
  field_media_image/target_id: fid
  field_media_image/alt: field_file_image_alt_text/0/value
  field_media_image/title: field_file_image_title_text/0/value

  # Description field.
  field_description: field_image_description
  # Caption field.
  field_caption: field_caption

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

To import local video, I add the URI scheme to the video migration.


# modules/custom/my_custom_module/migrations/my_media_local_video.yml
id: my_media_local_video
label: Files
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: video
  # See output of SELECT DISTINCT(SUBSTRING_INDEX(uri, ':', 1))  FROM file_managed WHERE type = 'video';
  scheme:
    - "public"
  constants:
    bundle: 'local_video'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # File field see media_entity.bundle.local_video.yml.
  field_media_video/target_id: fid
  # Title field.
  field_title: field_video_title
  # Transcript field.
  field_transcript: field_transcript

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

To import the youtube videos the URI was like youtube://v/video_id but embed field needs the youtube video URL so I created a process plugin to convert the video ID to URL.


# modules/custom/my_custom_module/migrations/my_media_video.yml
id: my_media_video
label: Files
migration_tags:
  - Custom

source:
  plugin: file_entity
  type: video
  # See output of SELECT DISTINCT(SUBSTRING_INDEX(uri, ':', 1))  FROM file_managed WHERE type = 'video';
  scheme:
    - "youtube"
  constants:
    bundle: 'video'

process:
  mid: fid
  bundle: 'constants/bundle'

  langcode:
    plugin: default_value
    source: language
    default_value: "und"

  name: filename

  uid:
    -
      plugin: skip_on_empty
      method: process
      source: uid
    -
      plugin: migration
      migration: my_users

  status: status
  created: timestamp
  changed: timestamp

  # Embed field see media_entity.bundle.video.yml.
  field_media_video_embed_field:
    plugin: youtube
    source: uri
  # Title field.
  field_title: field_video_title
  # Transcript field.
  field_transcript: field_transcript

destination:
  plugin: entity:media
  migration_dependencies:
    required:
      - my_files
      - my_users

The Youtube process plugin


<?php
# modules/custom/my_custom_module/src/Plugin/migrate/process/Youtube.php
namespace Drupal\my_module\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;

/**
 * Custom process plugin to convert youtube scheme uri to video url.
 *
 * @MigrateProcessPlugin(
 *   id = "youtube"
 * )
 */
class Youtube extends ProcessPluginBase {

  const SCHEME = 'youtube://';
  const BASE_URL = 'http://youtube.com/watch?';

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    // Convert youtube scheme uri to video url.
    if (strpos($value, static::SCHEME) !== FALSE) {
      $value = static::BASE_URL . implode('=', explode('/', str_replace(static::SCHEME, '', $value), 2));
    }
    else {
      $value = NULL;
    }
    return $value;
  }

}

I ran these migrations using migrate_drush.

Conclusion

On paper, file entity to media migration sounds difficult, but the migration API in Drupal core and its use of new entity and plugin APIs made the migration for a custom entity with complex fields a lot easier.

Thanks, to Lee Rowlands for writing the skip_youtube_files plugin. Also, thank you, Ben Dougherty, for all the advice.

The code can also be found as a gist.

Posted by Jibran Ijaz
Senior Drupal Developer

Dated

Comments

Comment by Lucas Hedding

Dated

Also very possible is to convert your entire file ecosystem into Media items by adding in extra ER relationships instead of the direct file relationship. Both when converting from d6 or d7 to D8.

Comment by Jibran Ijaz

Dated

Yes, but some files can't be treated as media for example compressed files.

Comment by Sunil

Dated

Hi Jibran,
Thanks for the tutorial, I ran this for migrating files and moving images to image media entity, but it is not working properly.
e.g,
- It is not creating thumbnail.
- there is no entry in media__image table.

So how to fix this
Thanks- Sunil

Pagination

Add new comment

Restricted HTML

  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd> <h2> <h3> <h4> <h5> <h6>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.
Not sure where to start? Try typing "hello" or "help" if you get stuck.