Description The sfMogileFSPlugin provides an interface to interact with a MogileFS tracker. Homepage http://www.danga.com/mogilefs Categories Media Backend

Leads

Name	Email
Benjamin Meynell	ude.odaroloc <<ta>> llenyemb

MogileFS Client. Current version is in SVN in the 1.2 branch. It's recommended you use that. Requires Symfony 1.2 and latest version of sfWebBrowserPlugin.

Releases for sf 1.0

Copyright (c) 2007 Benjamin Meynell

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Ticketing

Go to the ticketing system to view opened tickets or report a bug.

Join this plugin team

Other releases

Release 0.6.0 - 19/07/2007

Installation for sf 1.0

php symfony plugin-install http://plugins.symfony-project.org/sfMogileFSPlugin 

PEAR

Download the PEAR package

Repository

Go to the repository: http://svn.symfony-project.com/plugins/sfMogileFSPlugin

sfMogileFSPlugin

sfMogileFS enables symfony applications to interact with [MogileFS]. MogileFS is an anagram for "OMG Files" and was created by [http://www.livejournal.com LiveJournal] to handle the storage, replication and retrieval of the large amount of file uploads they were, and continue to, experience. Many of the web's most popular sites use MogileFS as their file store. This includes [http://www.digg.com Digg], [http://www.last.fm Last.fm], http://www.guba.com Guba, and others.

Prerequisites

The following must be installed and setup before using this plugin.

• Perlbal
• MogileFS
• CURL

How It Works

Perlbal is a load balancer that runs on port 80 with a pool of apache servers running behind it. In addition to dispatching requests to the internal servers, Perlbal also sends the critical X-REPROXY-URL header to MogileFS. MogileFS sends the file back based on the key given.

MogileFS Terminology

There are four main components to a MogileFS setup:

• tracker(s): talk to the MogileFS database. By default run on port 6001.
• database (a MySQL Cluster is recommended but simply one db server will work): maintains a list of storage nodes, files, etc.
• storage node(s). Physical disks (hard drives) where files are stored. Mogstored defaults to port 7500.
• client library (aka sfMogileFSPlugin) - talks to trackers (mogilefsd).

Storing files:

• domain: Top level separation of files. File keys are unique within domains. The Socialhouse.com domain (socialhouse) is defined within the socialhouse application in app.yml. A domain consists of a set of classes that define the files within the domain. class
• key: A unique textual string that identifies a file. Keys are unique within domains. Examples of keys: userpicture:34:39, phonepost:93:3834, userbackup:15.
• class: Part of exactly one domain. A class, in effect, only specifies the minimum replica count of a file. Examples of Socialhouse classes: image, thumbnail, video.
• minimum replica count: Property of a class. This defines how many times the files in that class need to be replicated onto different devices in order to ensure redundancy among the data and prevent loss.
• file: Every file is part of exactly one class. A file is a defined collection of bits uploaded to MogileFS to store. Files are replicated according to their minimum replica count. Each file has a key, is a part of one class, and is located in one domain.

Plugin Installation

$ php symfony plugin-install http://plugins.symfony-project.com/sfMogileFSPlugin

In your app.yml file add:

dev:
  sfMogileFSPlugin:
    domain:     mogilefs_domain
    timeout:    5
    trackers:   [222.222.222.2:6001](111.111.111.1:6001,)

You will also need to define settings for your production environment.

Modify your routing.yml (feel free to change "i", module, and action to whatever you'd like).

@mfs
  url:   /i/:mkey
  param: { module: file, action: showMogileFSFile }

This plugin was designed to primarily add and retrieve files from MogileFS. You will need to create your own media handling model(s), module(s) and action(s), though an example setup is given below.

Installation complete. Clear your cache.

Plugin Files

The plugin contains 4 classes and one helper. They are phpdocumented and unit tests are included in the tests/ folder.

• sfMogileFSBaseFile.class.php - contains methods shared by both sfMogileFSFile and sfMogileFSRemote file.
• sfMogileFSFile.class.php - represents a file before it's saved to MogileFS.
• sfMogileFSRemoteFile.class.php - represents a file retrieved from MogileFS.
• sfMogileFS.class.php - contains static methods for interacting with the MogileFS trackers.
• MogileFSAssetsHelper.php - wrapper for existing symfony helper's suchc as url_for and image_tag.

To output an image file from your template:

To add a file to MogileFS:

  $mfile = new sfMogileFSFile();
  $mfile->setKey($mfs_key);
  $mfile->setClass($mfs_class);
  $mfile->setFile($local_file);
  $mfile->save(); // saves to MogileFS

Alternatively:

$mfile = new sfMogileFSFile($key, $class, $file);
$mfile->save();

To retrieve a file directly from MogileFS as a string:

  $mfile = sfMogileFS::loadFile($mfs_key);

To get a valid tracker path:

  $mfile = sfMogileFS::loadFile($mfs_key);
  $mfile->getPath();

Example Setup

This is a complete example. However, You should only use this as a guide and should make changes based on your own application and needs.

The example assumes you already have a file in MogileFS with the key "cab9f3c712d04de874dafb0af0a0bf03e303e6e0".

schema.yml:

  files:
    _attributes:       { phpName: File }
    id:
    user_id:           { type: integer, foreignTable: users, foreignReference: id, onDelete: cascade }
    file_type_id:      { type: integer, foreignTable: file_types, foreignReference: id, onDelete: cascade }
    mime_type:                 varchar(128)
    extension:                 varchar(8)
    title:                     varchar(128)
    description:               varchar(255)
    created_at:
    updated_at:

  photos:
    _attributes:            { phpName: Photo }
    id:
    file_id:                { type: integer, foreignTable: files, foreignReference: id, onDelete: cascade }
    photo_type_id:          { type: integer, foreignTable: photo_types, foreignReference: id, onDelete: cascade }
    photo_status_type_id:   { type: integer, foreignTable: photo_status_types, foreignReference: id, onDelete: cascade }
    mogilefs_key:           { type: varchar(40), phpName: MogileFSKey, required: true }
    width:                          integer
    height:                         integer
    filesize:                       integer
    created_at:

routing.yml:

mfs:
  url:   /i/:mkey
  param: { module: file, action: showMogileFSFile }

/apps//modules/file/actions/uploadAction.class.php:

class uploadAction extends sfAction
{
  public function execute()
  {
    if ($this->getRequest()->getMethod() == sfRequest::POST && $this->getRequest()->hasFiles())
    {
      if ($this->getRequest()->hasFileErrors())
      {
        $this->setFlash('notice', 'An error occurred while uploading ('.$this->getRequest->getFileError('file').')');
        $this->redirect('@upload');
      }

      // save() method adds file to File and Photo tables as well as MogileFS
      $file = new File();
      $file->setUserId($this->getUser()->getId());
      $file->setFileTypeId(1); // image
      $file->setMimeType($this->getRequest()->getFileType('file'));
      $file->setExtension($this->getRequest()->getFileExtension('file'));
      $file->setTitle($this->getRequestParameter('title'));
      $file->setDescription($this->getRequestParameter('description'));
      $file->setTmpFile($this->getRequest()->getFilePath('file')); // not actually part of model but used by it
      $file->setFilesize($this->getRequest()->getFileSize('file')); // not actually part of the model but used by it
      $file->save();

      // delete upload
      unlink($file->getTmpFile());

      // redirect to upload form
      $this->setFlash('notice', 'Your file has been uploaded');
      $this->redirect('@upload');
    }
    else
    {
      // display upload form
      return sfView::SUCCESS;
    }
  }

//lib/model/Photo.php:

class Photo extends BasePhoto
{
  public function save($con = null)
  {
    if ($this->isNew())
    {
      $this->setMogileFSKey(hash('sha1', $this->getFileId().':'.$this->getId()));
    }
    parent::save($con);
  }
}

//lib/model/File.php:

class File extends BaseFile
{
  protected $filesize;
  protected $tmp_file;

  public function setFilesize($fileSize)
  {
    $this->filesize = $fileSize;
  }

  public function getFilesize()
  {
    return $this->filesize;
  }

  public function setTmpFile($filePath)
  {
    $this->tmp_file = $filePath;
  }

  public function getTmpFile()
  {
    return $this->tmp_file;
  }

  /**
   * save
   * adds original file to file table, sub-file (photo, etc) table and MogileFS
   * All steps are wrapped in a transaction.
   *
   * @param mixed $con
   * @access public
   * @return void
   */
  public function save($con = null)
  {
    if ($this->isNew())
    {
      list($width, $height) = getimagesize($this->getTmpFile());

      // store it
      $con = Propel::getConnection();
      try
      {
        $con->begin();

        // save to File table
        parent::save($con);

        // add to photo table
        if ($this->getFileTypeId() == 1)
        {
          $photo = new Photo();
          $photo->setFileId($this->getId()); // parent file
          $photo->setPhotoTypeId(1); // original photo
          $photo->setPhotoStatusTypeId(1); // uploaded but not converted
          $photo->setWidth($width);
          $photo->setHeight($height);
          $photo->setFilesize($this->getFilesize());
          $photo->save();
        }

        // save to MogileFS
        $sfMogileFSFile = new sfMogileFSFile();
        $sfMogileFSFile->setKey($photo->getMogileFSKey());
        $sfMogileFSFile->setClass('orig_photo');
        $sfMogileFSFile->setFile($this->getTmpFile());
        $sfMogileFSFile->setFilesize($photo->getFilesize());
        $sfMogileFSFile->save();

        $con->commit();
      }
      catch (Exception $e)
      {
        $con->rollback();
        throw $e;
      }
    }
  }
}

/apps//modules/file/actions/showMogileFSFileAction.class.php:

class showMogileFSFileAction extends sfAction
{
  public function execute()
  {
    // remove file's extension
    $mogileFSKey = preg_replace('@\.[\w\d]+$@', '', $this->getRequestParameter('mkey'));

    // get mogilefs key from photo table
    $c = new Criteria();
    $c->add(PhotoPeer::MOGILEFS_KEY, $mogileFSKey);
    $photo = PhotoPeer::doSelectOne($c);

    // load path from mogilefs
    $sfMogileFSRemoteFile = sfMogileFS::loadFile($mogileFSKey);

    // call perlbal's x-reproxy-url and set mime type
    $this->getResponse()->setHttpHeader('Content-type', $photo->getFile()->getMimeType(), true);
    $this->getResponse()->setHttpHeader('X-REPROXY-URL', $sfMogileFSRemoteFile->getPath(), true);

    return sfView::HEADER_ONLY;
  }
}

layout.php (or any template):

This results in the following HTML rendering:

<img src="/index.php/i/cab9f3c712d04de874dafb0af0a0bf03e303e6e0.jpg" />

Todo

• More detailed How It Works explanation
• Example of calling MogileFS using a separate php script and mod_rewrite (instead of having to launch symfony for every image call)
• Backend module with MogileFS stats, etc.

Changelog

trunk

2007-07-19 | 0.6.0 Alpha

• bmeynell: Initial release