Benchmark Review
: Comparison
between binary encoder vs. textual encoder
Study by OSS
Nokalva
If You Need Speed,
Go Binary
- Binary encoding is many times faster than textual
encoding
Recently, Erlang.org reported that their textual encoders for Megaco
are about 3 times faster than their binary encoders, and their text
decoders are 60% faster than their binary decoders. The report can be
found at www.erlang.org/project/megaco/encoding_comparison/index.html.
With this in mind, we set out to independently verify Erlang.org's findings,
and to determine whether such speed difference was specific to the Erlang.org
implementation, or if it was a function of how well the Erlang.org tools
are tuned for the textual encodings compared to binary encodings.
In
trying to reproduce the Erlang.org test results, we followed the same
criteria and procedures used by them. In addition to testing using the
Erlang tools, we ran the same test data through the OSS Nokalva's time-optimized
encoder/decoder (TOED). What we found was that the OSS software encoded/decoded
Megaco messages using the Packed Encoding Rules (PER) about 12 - 24
times faster than the Erlang.org software encoded messages in binary,
and about 4 - 8 times faster than the Megaco compact encodings (The
actual times were divided by 2 and by 4 to yield these factors, so as
to reflect claims that C is at most 2-4 times faster than Erlang). Furthermore,
OSS encoded the messages using the Basic Encoding Rules (BER) about
4 - 8 times faster than the Erlang.org software encoded messages in
the Megaco compact textual format.
Approaching
this review we had a number of questions, all boiling down to a common
theme: which are faster, binary or textual encoders? The answer is:
tools that are highly optimized for binary encodings are much faster
than those that are highly optimized for textual encodings. A secondary
conclusion that can be drawn from both the Erlang.org and OSS Nokalva
results is that for the Megaco ASN.1 specification, PER-encoded messages
are not only significantly smaller than messages encoded using BER,
but also take significantly less time to encode and decode.
For
both the OSS ASN.1 Tools and the Erlang tools, we ran tests to provide
timings of:
1) encode only
2) decode only
3) decode-then-encode which is the sum of decode only and encode only.
Output
of OSS measurement utility is available for download.
The
following is a summary of the performance tests for encode, decode,
decode-then-encode. In the graph, "erl_dist" refers to Erlang's
own textual encoding which is not used by Megaco, and "Erlang Compact"
refers to compact Megaco textual encodings. The PER and erl_dist encoding
results are just for comparison purposes, as Megaco mandates use of
BER for binary encodings, not PER, and the Megaco textual format for
textual encodings, not erl_dist.
The “faster
by” numbers shown are actual times, reflecting the speed of the
OSS encoder/decoder compared to the Erlang encoder/decoder. Since the
OSS runtime is written in C and the erlang.org runtime is written in
Erlang, and since it has been claimed that C is at most 2 to 4 times
faster than the native code build of Erlang, these numbers should be
divided by 2 and 4 in order to yield performance factors that take this
C/Erlang speed difference into consideration.
After testing and
review, we drew the following conclusions:
1. Tools that are optimized for binary encodings can perform these test
cases much faster than those highly optimized for textual encodings
presented here.
2. The OSS BER encoder/decoder is significantly faster than Erlang's
textual, BER and PER encoder/decoder. The performance difference is
even higher when OSS PER is used.
3. For Megaco, both OSS and Erlang encode and decode messages in PER
significantly faster than they do in BER.
4. For Megaco, the PER-encoded messages are significantly smaller than
either the BER-encoded or textually-encoded messages.
5. Erlang's textual encoders/decoders perform much better than their
BER and PER encoders because the textual encoders/decoders are much
more highly optimized than their BER/PER engines.
6. Our experience suggests that encoders and decoders for binary encoding
rules are faster than ones for textual encoding rules. However, any
claims as to which are faster cannot be substantiated without thoroughly
controlled, scientific analysis. This would require a great deal of
extensive research, not the meager testing that has been done here.
Research is needed which uses the same machine, same language, same
API, and the same runtime environment. Such evaluation should involve
equally competent teams of developers who have equal passion to write
the fastest and smallest encoding and decoding tools. Their results
would need to be carefully evaluated to determine that they had an equal
level of robustness in the face of exception data, and can handle high
numbers of transactions and execution over long periods of time.
The testing platform
for both tools was a machine configured as follows:
Intel Celeron 900MHz 256Mb RAM Red Hat Linux 7.1 kernel 2.4.16
Please refer to
the test setup for details on the
machine used and procedures.
If you would like to run your own evaluation, click
here for all necessary files and instructions.
View
uniGone Study
