+
MessagePack
+
+
MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON but it’s faster and smaller. For example, small integers (like flags or error code) are encoded into a single byte, and typical short strings only require an extra byte in addition to the strings themselves.
+
+
If you ever wished to use JSON for convenience (storing an image with metadata) but could not for technical reasons (binary data, size, speed…), MessagePack is a perfect replacement.
+
+
require 'msgpack'
+msg = [1,2,3].to_msgpack MessagePack.unpack(msg)
+
+
Add msgpack to your Gemfile to install with Bundler:
+
+
gem 'msgpack'
+
+
+
Or, use RubyGems to install:
+
+
gem install msgpack
+
+
+
Or, build msgpack-ruby and install from a checked-out msgpack-ruby repository:
+
+
bundle
+rake
+gem install --local pkg/msgpack
+
+
+
Use cases
+
-
+
Create REST API returing MessagePack using Rails + RABL
+ -
+
Store objects efficiently serialized by msgpack on memcached or Redis
+ -
+
In fact Redis supports msgpack in EVAL-scripts
+ -
+
Upload data in efficient format from mobile devices such as smartphones
+ -
+
MessagePack works on iPhone/iPad and Android. See also Objective-C and Java implementations
+ -
+
Design a portable protocol to communicate with embedded devices
+ -
+
Check also Fluentd which is a log collector which uses msgpack for the log format (they say it uses JSON but actually it’s msgpack, which is compatible with JSON)
+ -
+
Exchange objects between software components written in different languages
+ -
+
You’ll need a flexible but efficient format so that components exchange objects while keeping compatibility
+
+
+
Portability
+
+
MessagePack for Ruby should run on x86, ARM, PowerPC, SPARC and other CPU architectures.
+
+
And it works with MRI (CRuby) and Rubinius. Patches to improve portability are highly welcomed.
+
+
Serializing objects
+
+
Use MessagePack.pack
or to_msgpack
:
+
+
require 'msgpack'
+msg = MessagePack.pack(obj) msg = obj.to_msgpack
+File.binwrite('mydata.msgpack', msg)
+
+
+
Streaming serialization
+
+
Packer provides advanced API to serialize objects in streaming style:
+
+
pk = MessagePack::Packer.new(io)
+pk.(2).write(e1).write(e2).flush
+
+
+
See API reference for details.
+
+
Deserializing objects
+
+
Use MessagePack.unpack
:
+
+
require 'msgpack'
+msg = File.binread('mydata.msgpack')
+obj = MessagePack.unpack(msg)
+
+
+
Streaming deserialization
+
+
Unpacker provides advanced API to deserialize objects in streaming style:
+
+
u = MessagePack::Unpacker.new(io)
+u.each do |obj|
+ end
+
+
+
or event-driven style which works well with EventMachine:
+
+
def on_read(data)
+ @u ||= MessagePack::Unpacker.new
+ @u.feed_each(data) {|obj|
+ }
+end
+
+
+
See API reference for details.
+
+
Serializing and deserializing symbols
+
+
By default, symbols are serialized as strings:
+
+
packed = :symbol.to_msgpack MessagePack.unpack(packed)
+
+
This can be customized by registering an extension type for them:
+
+
MessagePack::DefaultFactory.register_type(0x00, Symbol)
+
+packed = :symbol.to_msgpack MessagePack.unpack(packed)
+
+
The extension type for symbols is configurable like any other extension type. For example, to customize how symbols are packed you can just redefine Symbol#to_msgpack_ext. Doing this gives you an option to prevent symbols from being serialized altogether by throwing an exception:
+
+
class Symbol
+ def to_msgpack_ext
+ raise "Serialization of symbols prohibited"
+ end
+end
+
+MessagePack::DefaultFactory.register_type(0x00, Symbol)
+
+[1, :symbol, 'string'].to_msgpack
+
+
Serializing and deserializing Time instances
+
+
There are the timestamp extension type in MessagePack, but it is not registered by default.
+
+
To map Ruby’s Time to MessagePack’s timestamp for the default factory:
+
+
MessagePack::DefaultFactory.register_type(
+ MessagePack::Timestamp::TYPE, Time,
+ packer: MessagePack::Time::Packer,
+ unpacker: MessagePack::Time::Unpacker
+)
+
+
+
See API reference for details.
+
+
Extension Types
+
+
Packer and Unpacker support Extension types of MessagePack.
+
+
pk = MessagePack::Packer.new(io)
+pk.register_type(0x01, MyClass1, :to_msgpack_ext) pk.register_type(0x02, MyClass2){|obj| obj.how_to_serialize() }
+uk = MessagePack::Unpacker.new()
+uk.register_type(0x01, MyClass1, :from_msgpack_ext)
+uk.register_type(0x02){|data| MyClass2.create_from_serialized_data(data) }
+
+
+
MessagePack::Factory
is to create packer and unpacker which have same extension types.
+
+
factory = MessagePack::Factory.new
+factory.register_type(0x01, MyClass1) factory.register_type(0x01, MyClass1, packer: :to_msgpack_ext, unpacker: :from_msgpack_ext)
+pk = factory.packer(options_for_packer)
+uk = factory.unpacker(options_for_unpacker)
+
+
+
For MessagePack.pack
and MessagePack.unpack
, default packer/unpacker refer MessagePack::DefaultFactory
. Call MessagePack::DefaultFactory.register_type
to enable types process globally.
+
+
MessagePack::DefaultFactory.register_type(0x03, MyClass3)
+MessagePack.unpack(data_with_ext_typeid_03)
+
+
Alternatively, extension types can call the packer or unpacker recursively to generate the extension data:
+
+
Point = Struct.new(:x, :y)
+factory = MessagePack::Factory.new
+factory.register_type(
+ 0x01,
+ Point,
+ packer: ->(point, packer) {
+ packer.write(point.x)
+ packer.write(point.y)
+ },
+ unpacker: ->(unpacker) {
+ x = unpacker.read
+ y = unpacker.read
+ Point.new(x, y)
+ },
+ recursive: true,
+)
+factory.load(factory.dump(Point.new(12, 34)))
+
+
Pooling
+
+
Creating Packer
and Unpacker
objects is expensive. For best performance it is preferable to re-use these objects.
+
+
MessagePack::Factory#pool
makes that easier:
+
+
factory = MessagePack::Factory.new
+factory.register_type(
+ 0x01,
+ Point,
+ packer: ->(point, packer) {
+ packer.write(point.x)
+ packer.write(point.y)
+ },
+ unpacker: ->(unpacker) {
+ x = unpacker.read
+ y = unpacker.read
+ Point.new(x, y)
+ },
+ recursive: true,
+)
+pool = factory.pool(5)
+pool.load(pool.dump(Point.new(12, 34)))
+
+
Buffer API
+
+
MessagePack for Ruby provides a buffer API so that you can read or write data by hand, not via Packer or Unpacker API.
+
+
This MessagePack::Buffer is backed with a fixed-length shared memory pool which is very fast for small data (<= 4KB), and has zero-copy capability which significantly affects performance to handle large binary data.
+
+
How to build and run tests
+
+
Before building msgpack, you need to install bundler and dependencies.
+
+
gem install bundler
+bundle install
+
+
+
Then, you can run the tasks as follows:
+
+
Build
+
+
bundle exec rake build
+
+
+
Run tests
+
+
bundle exec rake spec
+
+
+
Generating docs
+
+
bundle exec rake doc
+
+
+
How to build -java rubygems
+
+
To build -java gems for JRuby, run:
+
+
rake build:java
+
+
+
If this directory has Gemfile.lock (generated with MRI), remove it beforehand.
+
+
Updating documents
+
+
Online documentation (ruby.msgpack.org) is generated from the gh-pages branch. To update documents in gh-pages branch:
+
+
bundle exec rake doc
+git checkout gh-pages
+cp doc/* ./ -a
+
+
+
Copyright
+
-
+
Author
+ -
+
Sadayuki Furuhashi frsyuki@gmail.com
+ -
+
Copyright
+ -
+
Copyright © 2008-2015 Sadayuki Furuhashi
+ -
+
License
+ -
+
Apache License, Version 2.0
+
+