Benchmarks and Test Methodologies
BYTE rates products based on technology, implementation, performance, features, and value. We depend on the following benchmarks because of their applicability to the ta
sk, ability to produce relevant metrics, and reliability.
Release 2 of BYTE Magazine's BYTEmark benchmark program (previously
known as BYTE's Native Mode Benchmarks) is designed to test raw CPU
speed across a variety of platforms. The BYTEmark documentation
discusses the tests at the algorithm level.
BYTEmark is designed to expose the capabilities of a system's CPU, FPU,
and memory system.
To measure a system's real-world performance when running
typical business applications, we use
BAPCo's SYSmark 32 for Windows 95 and Windows NT ver 1.0.
This benchmark suite
comprises the retail versions of eight application programs and
measures the speed with which the system under test executes
pre-determined scripts of user tasks typically performed when using
these applications.
All programs run in native 32-bit mode and each application script
is performed three times. The performance times of the individual
applications are weighted and combined into both category-based
performance scores as well as a single overall score.
The application programs employed by SYSmark 32 are:
- WORD PROCESSING
- Microsoft Word 7.0
- Lotus WordPro 96
- SPREADSHEET
- DATABASE
- DESKTOP GRAPHICS
- DESKTOP PRESENTATION
- Lotus Freelance Graphics 96
- Microsoft Powerpoint 7.0
- DESKTOP PUBLISHING
IMPORTANT NOTE: The BYTE/SYSmark Benchmark Disk
If you are experiencing problems using the BYTE/SYSmark Benchmark Disk, please go to
http://www.bapco.com/misc.htm
, scroll down to the "SYSmark32" section and read "Symptom #3" and the solution. If this information is insufficient to resolve your problems, you can reach BAPCO at 1-800-321-0457.
To compare the performance of cross-platform systems running common
business applications exclusively under Windows NT, BYTE has used
BAPCo SYSmark for Windows NT.
This
benchmark suite comprises five application programs and measures
the speed with which the system under test executes pre-determined
Scripts of user tasks typically performed when using these
applications.
The application programs used are:
- Word Processing:
Microsoft Word 6.0
- Spreadsheet:
Microsoft Excel 5.0
- Presentation Graphics:
Microsoft PowerPoint 4.0
- Computer-Aided Design:
OrCAD Layout for Windows 7.0
- Project Management:
Welcom Software Texim Project 2.0e
All programs run in native 32-bit mode except PowerPoint, which runs
the application code in 16-bit emulation mode.
Makers of high-end 3D graphics accelerators typically quote
performance as the number of primitives (such as triangles) drawn
per second. Numbers that, in the absence of additional information
such as the context, size, shading, color depth, and the smoothing
method used to draw those triangles, make direct comparisons
difficult.
Given the growing importance of OpenGL on the desktop,
we characterize 3D performance using the Viewperf benchmark,
developed by the Standard Performance Evaluatio
n Corporation (SPEC)
Graphics Performance Characterization (GPC) Group's
OpenGL Performance Characterization (OPC) Project.
Viewperf is a portable benchmark and the
current industry standard for evaluating OpenGL performance.
Viewperf does not benchmark individual primitives, but measures how
well a system accelerates actual applications data sets called
viewsets.
- The CDRS viewset is derived from Parametric Technology's
modeling and rendering software for computer-aided industrial
design. The test measures seven different operations on a model of
a lawnmower.
- The DX viewset is based on IBM's Visualization Data
Explorer, a general purpose scientific data visualization and
analysis package. The DX benchmark draws a set of particle traces
thorough a vector flow field.
Viewperf measures frames-per-second for each component test.
The single result for a each viewset
is a weighted geometric mean.
Intel Media Benchmark
To measure MMX performance, BYTE uses the
Intel Media Benchmark
that puts the systems through a gauntlet of
multimedia tasks. The algorithms used are coded for both scalar and
MMX-enabled processors.
The automated test suite simulates the following activities that may exploit
MMX-enabled CPUs.
- Home Photo Editing with
Adobe Photo Deluxe
- Intel Indeo multimedia video clip performance
- MPEG-1 video file performance
- Direct 3D gaming
- Business Photo Manipulation with
Adobe Photoshop
NSTL InterMark
NSTL's
InterMark benchmark tool profiles applications by
capturing the system calls and actions. InterMark then replays
these actions to test a system's components witho
ut the need for
third-party applications. A precision event timer is used to
measure the response time of the system for each task. NSTL
utilizes the video, hard disk, CPU, and CD-ROM components of the
InterMark suite to test these Pentium MMX PCs.
Video tests are broken into three categories: Windows draw,
bit-mapped images, and non-bit-mapped images. The Windows draw
tests use the Windows API calls to draw images, including text, a
picture in the Windows Metafile format, horizontal and vertical
lines, rectangles, and ellipses. The bit-mapped images and non bit-
mapped image tests display an image on the screen. This image is
then stretched to one and one-half, and then twice its original
size.
Hard Disk tests measure the speed of the hard disk for reading
and writing. The test simulates sequential, constant rate
sequential, random, localized random, and segmented activity in
varying block sizes. The tests measure the average response time,
the sustained rate, the burst rate, the r
andom access time, and CPU
utilization of the hard disk.
CPU tests measure the performance of the processor for both
integer and floating-point calculations. These tests provide an
index of the computing power of the processor chip. CD-ROM tests
examine several different aspects of CD-ROM performance: random
service time at 500 and 1,000 milliseconds, cached service time at
13.3 milliseconds; and CPU utilization at 550KB per second.
BYTE Benchmarks