Prewarm your EBS backed EC2 MySQL slaves

This is the story of cold blocks and mismatched instances and how they will cause you pain and cost you money until you understand why.

Most of the clients that we support run on the Amazon cloud using either RDS or running MySQL on plain EC2 instances using (Provisioned IOPS) PIOPS EBS for data storage.

As expected the common architecture is running a master with one or more slaves handling the read traffic.

A common problem is that after the slaves are provisioned (normally created from an EBS snapshot) they lag badly due to slow IO performance.

Unfortunately what tends to be lost in the “speed of provisioning new resources” fetish is some limitations in terms of data persistence layer (EBS).

If you are using EBS and you have created the EBS volume from snapshot or created a new volume you have to pre-warm the EBS volume otherwise you will suffer a bad (I mean seriously bad) first usage penalty.  Bad? I am talking up to 50% performance drop[1]. So that expensive PIOPS EBS volume you created is going to perform like rubbish every time it reads/writes a cold block.

The other thing which also tends to happen is mixing up the wrong instance (network performance) with the PIOPS EBS. This the classic networked storage, the network is the bottleneck. If your instance type has limited network performance, having a higher PIOPS than the network can handle means you are wasting money (on PIOPS) you can’t use. A bit like in the old days (of dedicated servers and SAN storage) where the SAN could deliver 200-300Mbytes per sec, but the 1 Gigabit network could only do 40-50Mbytes per sec.

Here is the real downside, using the cloud you can provision new resources to handle peak load (in the case more MySQL slaves to handle read load) as fast as you can click, or faster using API calls, or even automagically, if you have some algo forecast the need for additional resources. But… the EBS is all cold blocks, so these new instances will be up and available in minutes but the IO performance will be poor until you either pre-warm or the slave gets around to writing/reading all blocks.

So the common solution is to pre-warm the blocks using dd to read the EBS device (and warm the block) to /dev/null

eg: sudo dd if=/dev/xvdf of=/dev/null bs=1M

Consider how long this will take for any reasonable sized DB (200GBytes) using an instance with 1 Gigabit network.

200Gigabytes read at 50Mbytes/sec  = 200,000 Mbytes/50 = 4000 secs = 3600 (1hr) + 400 (6 mins 40 secs) =~ more than 1 hr.

So you or your algo provisioned a new EC2 instance for the database in minutes but either your IO will be rubbish for an extended period, or you wait more than 1 hr per 200GB to have the EBS pre-warmed.

What are the solutions?

  1. Forecast further in advance depending on the size of your db (or any other persistent storage layer eg NoSQL etc)
  2. Use ephemeral storage and manage the increased risk of data loss in the event of instance termination.
  3. Break your DB or your application into smaller pieces aka micro services.[2]
  4. Pay more $ and have your databases stay around longer so waiting for a instance to be ready in the beginning is not a problem.

As you can expect, most businesses are happy with option 4. Pay more, leave instances around like they were dedicated servers (base load). Amazon is happy too.

Option 3 whilst requiring some thought (argh) and additional complexity is where the real speed of provisioning, dare I say it, agile nature of the cloud will bear the most fruit.

[1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html

[2] http://martinfowler.com/articles/microservices.html

 

Oracle RAC on EC2 redux

I was reading some RSS feeds the other day and noticed that Jeremy Schneider over at Ardent Performance Computing was working on getting Oracle RAC working on Amazon EC2.
http://www.ardentperf.com/2011/03/04/byo-oracle-rac-on-ec2/
He looks to have solved the whole Virtual IP issue by using another instance. Nice solution!

When I get a spare moment (don’t believe for a minute that the lack of posts means I am not busy) it would be good to take the scripts and get the whole Oracle RAC working in Amazon EC2 finally!

Other News:

I have had the chance to play with some columnar databases, Vertica and Ingres VectorWise and the performance is good. I used the TPC-H benchmark to test to a scale 20 (small only due to a lack of disk space). So the results are nothing like the recent Scale 100 benchmarks that Ingres VectorWise did but useful nonetheless. Sadly I can’t publish any scripts or results as the IP is owned by my current employer.

Currently I am focused on improving my skills in predictive modeling and analytics. This is using the data rather than just supporting/recovering and hand-holding the data a.k.a. being a DBA.

Listening to trance, in the zone and most definitely Having Fun!

Paul

OurDelta MySQL on EC2 – install

Summary:

Arjen would give me an earful if I got this wrong or poorly worded.

“OurDelta produces enhanced builds for MySQL, with OurDelta and third-party patches, for common production platforms” from http://ourdelta.org/about

Over the next series of articles I am going to put the many additions to the MySQL 5.0 baseline through their paces on Amazon EC2.

Using a base CentOS 4.4 I had lying around on Amazon S3 I had Ourdelta installed and secure in no time flat.

This is outline for the installing Ourdelta onto CentOS 4

Install:


yum install yum-plugin-protectbase
rpm --import http://mirror.ourdelta.org/deb/ourdelta.gpg
rpm -Uvh http://mirror.ourdelta.org/yum/CentOS/OurDelta-release-0.0.4-1.noarch.rpm
mkdir downloads
cd downloads/
wget http://mirror.ourdelta.org/yum/CentOS/4/i386/RPMS/MySQL-OurDelta-server-5.0.67.d7-44.el4_7.i386.rpm
wget http://mirror.ourdelta.org/yum/CentOS/4/i386/RPMS/MySQL-OurDelta-shared-5.0.67.d7-44.el4_7.i386.rpm
wget http://mirror.ourdelta.org/yum/CentOS/4/i386/RPMS/MySQL-OurDelta-test-5.0.67.d7-44.el4_7.i386.rpm
wget http://mirror.ourdelta.org/yum/CentOS/4/i386/RPMS/MySQL-OurDelta-client-5.0.67.d7-44.el4_7.i386.rpm
wget http://mirror.ourdelta.org/yum/CentOS/4/i386/RPMS/MySQL-OurDelta-devel-5.0.67.d7-44.el4_7.i386.rpm
yum localinstall MySQL-OurDelta-*
/usr/bin/mysql_secure_installation
ps -ef|grep mysql
mysql -u root -p

Comments:

Thats it. Straightforward install.
I am getting lazy in my old age I used a pipe combo to generate the install commands using this:

history |grep -v history|awk ‘{ print $2″ “$3” “$4 }’

The next article will cover the upgrade of any existing MySQL 5.0 release to OurDelta.
I have saved this Ourdelta MySQL install as an AMI.

If people are interested in making this a public AMI please post a comment.

Have Fun

Paul

Sysbench fileio vs small EC2 with EBS

Background:

I got a request from a reader to repeat the sysbench fileio benchmarks against the new Amazon EBS (Elastic Block Store).

As mentioned elsewhere as this is similar to network attached storage (NAS) it will be constrained by the speed of network.

The past series on sysbench fileio on EC2

Sysbench fileio vs EC2 Part 1
Sysbench fileio vs Large EC2 Part 2
seeker io benchmark vs EC2

Results:

As expected the network (1 Gigabit) is the upper limit of throughput (MB/s).

What this means is the local disk (not /mnt) is still the fastest disk available in terms of requests per sec and throughput (MB/s).

Adding an addition EBS volume as a RAID 0 was marginally better for single threaded reads but for many threads both the read performance was better.
The write performance was much better for single and many threads.

The additional flexibility of being able to attach and detach volumes, take snapshots is the big plus for using EBS.

At this point I made no attempt to optimize the RAID 0 chunksize. It would require more work, however at the end of the day, 90-100 MB/s is maximum.

How to use:

Same as before, stick your intensive IO portions on / and anything readonly on either /mnt or on Amazon EBS.

I am hanging out for an instance which provides infiniband network speeds or better. A bunch of clustering technology (shared nothing and shared disk) rely on good network bandwidth.

Have Fun

p.s. I had written this up about a month and half ago however delayed publishing as it isn’t a great result i.e. network disk is constrained by network bandwidth.

How to use EC2 mountpoints

It has been quiet around here. My workload has been high and I have been spending my down time reading, rather than experimenting and hence publishing.

Frederic commented in a recent post about Sysbench fileio benchmark.
If you want to see whole set of posts use EC2 Benchmark

There were three questions and my reply was going to be long. So now it is a post.

Questions:

  1. /dev/sda/ : do you mean that it would be better to separate the tables of the database or simply use /dev/sda (if 10GB is enough for DB storage)
  2. I found info about db caching (http://dev.mysql.com/doc/refman/5.0/en/query-cache.html), but don’t know what you are talking about file caching. Is there any link you could share about this principle.
  3. what kind of filesystem do you use on /mnt and what is the size of the partition created.

Answers:

  1. Given what the IO benchmarks are saying when they bypass the OS caching mechanism i.e. /dev/sda is faster than /dev/sdb for writes. Use /dev/sda for your binlogs and anything you want to have fast write access. Many databases are 95-98% reads so putting them on the /dev/sdb (/mnt) with OS caching is going to fine.
  2. There are many levels of caching. Google for Facebook MySQL presentation. Many websites which need to scale use Memcache upfront of the database. This means that most reads from the database are served from Memcache. MySQL also has a query cache as you found. File caching is just OS caching. This is normally on by default on most linux flavours. Essentially there is no point in wasting any free RAM, so the OS will cache any physical reads so that next time they can be served from memory rather than disk.
  3. I was just using the standard ext3 file system. You can run Logical Volumes on top of it. There are people who will argue for the pros and cons. It was more just what I was familiar with at the time. I was not stressing about the file system type, more about the difference in performance of the two virtual storages available.

Have Fun

Paul

Sysbench fileio vs XLarge EC2 Part 3

Background:

If you have been following the series. I have gone back to checking the raw IO performance of all the Amazon EC2 instances.
Sysbench fileio vs EC2 Part 1
Sysbench fileio vs Large EC2 Part 2
seeker io benchmark vs EC2

In the meantime Amazon has finally announced that they are providing persistent storage for all EC2 instances. This has removed a bunch of complexity from running a database on any EC2 instance.

Results:

The percentage improvement of the extra large instance over the large instance was pretty minor. It is still much faster than the small instance. There was some better results for the higher threads, more files random reads and random writes.

Comparison to seeker benchmark:

Seeker benchmark got the following for the extra large instance:

/ (/dev/sda1) : Results: 1547 seeks/second, 0.65 ms random access time
/mnt (/dev/sdb): Results: 1167 seeks/second, 0.86 ms random access time

The seeker benchmark were again widely different when there was no space used on /mnt
compared to when there was.
The Xen virtual machine is not cheating, but it is not showing a realistic result when /mnt (/dev/sdb) is empty.

Comments:

All the results were double-checked by running iostat -x whilst the benchmarks were running. Confirming the results for both sysbench and seeker.

My recommendation, based on these results remains the same as before:

  1. Place files which require good raw read and write performance (without cache) on /dev/sda
  2. Use linux file cache or database caching for most reads. But you know that already right.
  3. Don’t rely on good read or write performance without caching on /mnt.
  4. iostat is your friend. You can check your requests/sec, wait queues and disk utilization on your application, not just a benchmark tool.

If/When I get accepted to the persistence storage beta, I will rerun these tests again. It should be interesting.

Raw Results:



Specs EC2 xlarge instance

[root@ip-10-251-146-159 sysbench-0.4.8]# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 270
stepping : 2
cpu MHz : 2004.542
cache size : 1024 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 5013.04
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 270
stepping : 2
cpu MHz : 2004.542
cache size : 1024 KB
physical id : 1
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 5013.04
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 2
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 270
stepping : 2
cpu MHz : 2004.542
cache size : 1024 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 5013.04
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 270
stepping : 2
cpu MHz : 2004.542
cache size : 1024 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 5013.04
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

[root@ip-10-251-146-159 sysbench-0.4.8]# free
total used free shared buffers cached
Mem: 15731980 581388 15150592 0 12724 217576
-/+ buffers/cache: 351088 15380892
Swap: 0 0 0
[root@ip-10-251-146-159 sysbench-0.4.8]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.9G 1.2G 8.3G 12% /
/dev/sdb 414G 199M 393G 1% /mnt
[root@ip-10-251-146-159 sysbench-0.4.8]# uname -a
Linux ip-10-251-146-159 2.6.20-1.3002.fc6xen #1 SMP Mon Aug 13 14:21:21 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
[root@ip-10-251-146-159 sysbench-0.4.8]# cat /proc/meminfo
MemTotal: 15731980 kB
MemFree: 15150832 kB
Buffers: 12752 kB
Cached: 217624 kB
SwapCached: 0 kB
Active: 163920 kB
Inactive: 71036 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 4568 kB
Mapped: 3464 kB
Slab: 19628 kB
SReclaimable: 11984 kB
SUnreclaim: 7644 kB
PageTables: 832 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 7865988 kB
Committed_AS: 14628 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 68 kB
VmallocChunk: 34359738299 kB
[root@ip-10-251-146-159 sysbench-0.4.8]#

sysbench_fileio.sh

[root@ip-10-251-146-159 ~]# ./sysbench_fileio.sh
sysbench v0.4.8: multi-threaded system evaluation benchmark

1 files, 131072Kb each, 128Mb total
Creating files for the test...
sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 16384
1 files, 128Mb each
128Mb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed: 767647 Read, 0 Write, 0 Other = 767647 Total
Read 11.713Gb Written 0b Total transferred 11.713Gb (199.91Mb/sec)
12794.04 Requests/sec executed

Test execution summary:
total time: 60.0004s
total number of events: 767647
total time taken by event execution: 58.0262
per-request statistics:
min: 0.0001s
avg: 0.0001s
max: 0.0391s
approx. 95 percentile: 0.0001s

Threads fairness:
events (avg/stddev): 767647.0000/0.00
execution time (avg/stddev): 58.0262/0.00

sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64

Extra file open flags: 16384
1 files, 128Mb each
128Mb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Done.

Operations performed: 1000000 Read, 0 Write, 0 Other = 1000000 Total
Read 15.259Gb Written 0b Total transferred 15.259Gb (384.1Mb/sec)
24582.41 Requests/sec executed

Test execution summary:
total time: 40.6795s
total number of events: 1000000
total time taken by event execution: 2600.3658
per-request statistics:
min: 0.0002s
avg: 0.0026s
max: 0.5825s
approx. 95 percentile: 0.0027s

Threads fairness:
events (avg/stddev): 15625.0000/2.68
execution time (avg/stddev): 40.6307/0.00

sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 16384
1 files, 128Mb each
128Mb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed: 0 Read, 593313 Write, 0 Other = 593313 Total
Read 0b Written 9.0532Gb Total transferred 9.0532Gb (154.51Mb/sec)
9888.47 Requests/sec executed

Test execution summary:
total time: 60.0005s
total number of events: 593313
total time taken by event execution: 58.4663
per-request statistics:
min: 0.0001s
avg: 0.0001s
max: 4.8524s
approx. 95 percentile: 0.0001s

Threads fairness:
events (avg/stddev): 593313.0000/0.00
execution time (avg/stddev): 58.4663/0.00

sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64

Extra file open flags: 16384
1 files, 128Mb each
128Mb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Threads started!
Time limit exceeded, exiting...
(last message repeated 63 times)
Done.

Operations performed: 0 Read, 525171 Write, 0 Other = 525171 Total
Read 0b Written 8.0135Gb Total transferred 8.0135Gb (127.25Mb/sec)
8144.28 Requests/sec executed

Test execution summary:
total time: 64.4834s
total number of events: 525171
total time taken by event execution: 4124.9270
per-request statistics:
min: 0.0001s
avg: 0.0079s
max: 64.4652s
approx. 95 percentile: 0.0001s

Threads fairness:
events (avg/stddev): 8205.7969/17156.01
execution time (avg/stddev): 64.4520/0.04

sysbench v0.4.8: multi-threaded system evaluation benchmark

Removing test files...
sysbench v0.4.8: multi-threaded system evaluation benchmark

128 files, 1024Kb each, 128Mb total
Creating files for the test...
sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64

Extra file open flags: 16384
128 files, 1Mb each
128Mb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Threads started!
Time limit exceeded, exiting...
(last message repeated 63 times)
Done.

Operations performed: 0 Read, 951836 Write, 0 Other = 951836 Total
Read 0b Written 14.524Gb Total transferred 14.524Gb (247.86Mb/sec)
15862.76 Requests/sec executed

Test execution summary:
total time: 60.0044s
total number of events: 951836
total time taken by event execution: 3837.2266
per-request statistics:
min: 0.0004s
avg: 0.0040s
max: 6.8536s
approx. 95 percentile: 0.0058s

Threads fairness:
events (avg/stddev): 14872.4375/49.38
execution time (avg/stddev): 59.9567/0.00

sysbench v0.4.8: multi-threaded system evaluation benchmark

128 files, 16384Kb each, 2048Mb total
Creating files for the test...
sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 16384
128 files, 16Mb each
2Gb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed: 5029 Read, 0 Write, 0 Other = 5029 Total
Read 78.578Mb Written 0b Total transferred 78.578Mb (1.3095Mb/sec)
83.81 Requests/sec executed

Test execution summary:
total time: 60.0043s
total number of events: 5029
total time taken by event execution: 59.9890
per-request statistics:
min: 0.0001s
avg: 0.0119s
max: 1.2131s
approx. 95 percentile: 0.0361s

Threads fairness:
events (avg/stddev): 5029.0000/0.00
execution time (avg/stddev): 59.9890/0.00

sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64

Extra file open flags: 16384
128 files, 16Mb each
2Gb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Time limit exceeded, exiting...
(last message repeated 63 times)
Done.

Operations performed: 14103 Read, 0 Write, 0 Other = 14103 Total
Read 220.36Mb Written 0b Total transferred 220.36Mb (3.6463Mb/sec)
233.36 Requests/sec executed

Test execution summary:
total time: 60.4337s
total number of events: 14103
total time taken by event execution: 3854.5087
per-request statistics:
min: 0.0003s
avg: 0.2733s
max: 0.9353s
approx. 95 percentile: 0.5471s

Threads fairness:
events (avg/stddev): 220.3594/1.71
execution time (avg/stddev): 60.2267/0.13

sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 16384
128 files, 16Mb each
2Gb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed: 0 Read, 33598 Write, 0 Other = 33598 Total
Read 0b Written 524.97Mb Total transferred 524.97Mb (8.7416Mb/sec)
559.46 Requests/sec executed

Test execution summary:
total time: 60.0542s
total number of events: 33598
total time taken by event execution: 59.9642
per-request statistics:
min: 0.0001s
avg: 0.0018s
max: 6.7853s
approx. 95 percentile: 0.0001s

Threads fairness:
events (avg/stddev): 33598.0000/0.00
execution time (avg/stddev): 59.9642/0.00

sysbench v0.4.8: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64

Extra file open flags: 16384
128 files, 16Mb each
2Gb total file size
Block size 16Kb
Number of random requests for random IO: 1000000
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random write test
Threads started!
Time limit exceeded, exiting...

(last message repeated 63 times)
Done.

Operations performed: 0 Read, 226 Write, 0 Other = 226 Total
Read 0b Written 3.5312Mb Total transferred 3.5312Mb (46.928Kb/sec)
2.93 Requests/sec executed

Test execution summary:
total time: 77.0549s
total number of events: 226
total time taken by event execution: 4365.1668
per-request statistics:
min: 0.7881s
avg: 19.3149s
max: 56.0941s
approx. 95 percentile: 45.2706s

Threads fairness:
events (avg/stddev): 3.5312/0.85
execution time (avg/stddev): 68.2057/5.70

sysbench v0.4.8: multi-threaded system evaluation benchmark



iostat -x


[root@ip-10-251-146-159 ~]# iostat -x 1 |egrep "sda1|sdb"
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sdb 0.04 0.00 0.05 0.01 0.77 0.08 13.80 0.00 5.31 4.00 0.02
sda1 2.13 39.92 5.01 7.66 194.36 380.66 45.37 0.11 8.59 0.68 0.86
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 9.00 0.00 2.00 0.00 88.00 44.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 15.84 29500.00 1162.38 2992.08 37196.04 259936.63 71.52 83.01 19.98 0.16 65.35
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 88.89 0.00 12263.64 0.00 391886.87 0.00 31.96 0.75 0.06 0.06 75.15
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 75.25 0.00 12295.05 0.00 393188.12 0.00 31.98 0.67 0.05 0.05 66.93
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 83.17 0.00 11434.65 0.00 365592.08 0.00 31.97 0.72 0.06 0.06 71.68
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 74.75 0.00 11310.10 0.00 361632.32 0.00 31.97 0.69 0.06 0.06 69.09
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 84.85 0.00 11898.99 0.00 380412.12 0.00 31.97 0.73 0.06 0.06 72.73
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 100.00 7.00 12185.00 3.00 389640.00 80.00 31.98 0.76 0.06 0.06 76.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 95.05 0.00 12287.13 0.00 392776.24 0.00 31.97 0.71 0.06 0.06 70.89
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 108.00 0.00 12829.00 0.00 410080.00 0.00 31.97 0.75 0.06 0.06 74.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 100.00 0.00 12552.00 0.00 401216.00 0.00 31.96 0.72 0.06 0.06 72.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 105.05 0.00 12593.94 0.00 402650.51 0.00 31.97 0.77 0.06 0.06 77.17
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 73.27 0.99 12486.14 1.98 399112.87 23.76 31.96 0.71 0.06 0.06 71.29
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 103.03 0.00 12582.83 0.00 402262.63 0.00 31.97 0.74 0.06 0.06 73.54
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 75.25 0.00 12286.14 0.00 392681.19 0.00 31.96 0.80 0.07 0.07 80.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 84.00 0.00 12086.00 0.00 386304.00 0.00 31.96 0.84 0.07 0.07 82.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 97.03 0.00 11920.79 0.00 381021.78 0.00 31.96 0.78 0.07 0.07 78.42
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 106.00 0.00 12314.00 0.00 393760.00 0.00 31.98 0.69 0.06 0.06 68.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 78.79 1.01 12480.81 2.02 398965.66 24.24 31.96 0.72 0.06 0.06 71.52
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 98.00 0.00 12184.00 0.00 389536.00 0.00 31.97 0.77 0.06 0.06 76.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 104.00 0.00 12159.00 0.00 388672.00 0.00 31.97 0.80 0.07 0.06 78.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 101.01 0.00 11946.46 0.00 381963.64 0.00 31.97 0.76 0.06 0.06 76.36
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 92.93 0.00 11934.34 0.00 381575.76 0.00 31.97 0.75 0.06 0.06 74.75
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 88.24 0.00 11488.24 0.00 367184.31 0.00 31.96 0.78 0.07 0.07 77.25
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 78.79 1.01 11486.87 2.02 367321.21 24.24 31.97 0.86 0.07 0.07 85.66
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 93.07 0.00 11892.08 0.00 380039.60 0.00 31.96 0.79 0.07 0.07 78.81
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 92.93 0.00 12095.96 0.00 386585.86 0.00 31.96 0.79 0.07 0.06 78.38
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 89.11 0.00 12023.76 0.00 384285.15 0.00 31.96 0.80 0.07 0.07 80.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 76.77 0.00 12563.64 0.00 401583.84 0.00 31.96 0.83 0.07 0.07 82.83
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 104.95 0.00 12185.15 0.00 389607.92 0.00 31.97 0.82 0.07 0.07 82.38
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 96.00 10.00 12295.00 45.00 393088.00 440.00 31.89 0.73 0.06 0.06 72.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 128.00 0.00 14030.00 0.00 448480.00 0.00 31.97 0.69 0.05 0.05 68.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 110.89 0.00 13545.54 0.00 433013.86 0.00 31.97 0.71 0.05 0.05 71.29
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 117.17 0.00 13342.42 0.00 426408.08 0.00 31.96 0.69 0.05 0.05 68.69
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 92.93 0.00 13596.97 0.00 434715.15 0.00 31.97 0.66 0.05 0.05 65.86
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 79.21 0.00 13179.21 0.00 421196.04 0.00 31.96 0.70 0.05 0.05 69.70
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 114.00 1.00 14216.00 2.00 454560.00 24.00 31.97 0.70 0.05 0.05 70.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 112.00 0.00 13892.00 0.00 444128.00 0.00 31.97 0.72 0.05 0.05 71.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 108.00 0.00 13439.00 0.00 429568.00 0.00 31.96 0.60 0.04 0.04 59.60

iostat -x at five second intervals

[root@ip-10-251-146-159 ~]# iostat -x 5 |egrep "sda1|sdb"
sdb 0.04 0.00 0.05 0.01 0.73 0.08 13.80 0.00 5.31 4.00 0.02
sda1 6.14 66.66 540.68 10.24 17316.42 615.22 32.55 0.22 0.39 0.07 4.02
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 100.80 2.00 12478.00 0.40 398976.00 19.20 31.97 0.74 0.06 0.06 74.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 111.60 0.20 14267.40 0.40 456108.80 4.80 31.97 0.64 0.05 0.05 64.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 116.60 0.00 14198.80 0.00 453900.80 0.00 31.97 0.74 0.05 0.05 74.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 257.49 0.40 23116.97 1.40 741703.79 14.37 32.08 55.29 2.39 0.04 96.69
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 281.96 1.60 25049.90 0.40 803738.68 16.03 32.09 63.47 2.53 0.04 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 252.00 0.20 22304.40 0.40 715628.80 4.80 32.08 63.48 2.85 0.04 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 274.65 0.20 23675.65 0.40 759632.73 4.79 32.08 63.29 2.67 0.04 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 287.80 0.20 24654.80 0.40 791142.40 4.80 32.09 63.29 2.57 0.04 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 286.40 4.60 24286.40 1.60 779385.60 49.60 32.09 63.37 2.61 0.04 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 304.60 0.20 25854.40 0.40 829440.00 4.80 32.08 63.36 2.45 0.04 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 294.00 0.00 25450.40 0.20 816780.80 1.60 32.09 63.30 2.49 0.04 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 76.40 0.80 6545.80 5196.00 210112.00 164884.80 31.94 17.08 1.44 0.07 87.92
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.40 0.00 6342.20 0.00 201396.80 31.76 0.88 0.18 0.14 87.92
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 12910.42 0.00 409373.95 31.71 0.74 0.06 0.06 73.91
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 12959.40 0.00 411294.40 31.74 0.73 0.06 0.06 72.64
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 13295.00 0.00 421681.60 31.72 0.73 0.05 0.05 71.44
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 13015.80 0.00 412952.00 31.73 0.75 0.06 0.06 74.56
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10343.51 0.00 328059.08 31.72 0.82 0.08 0.08 80.48
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.60 0.00 4327.25 0.00 137195.99 31.71 1.62 0.08 0.21 91.06
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 6437.33 0.00 204322.55 31.74 2.03 0.51 0.13 84.87
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10635.80 0.00 337425.60 31.73 0.79 0.07 0.07 78.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.60 0.00 10701.00 0.00 339233.67 31.70 0.78 0.07 0.07 78.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10719.96 0.00 340003.19 31.72 0.77 0.07 0.07 76.89
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.60 0.00 10713.80 0.00 340265.60 31.76 0.77 0.07 0.07 76.56
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10963.73 0.00 347836.47 31.73 0.80 0.07 0.07 79.68
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2322.55 0.00 73606.39 31.69 0.95 0.05 0.41 95.17
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.60 0.00 5976.25 0.00 189649.50 31.73 0.85 0.28 0.14 84.55
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 13295.60 0.00 421867.20 31.73 0.75 0.06 0.06 74.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 12889.00 0.00 408760.00 31.71 0.73 0.06 0.06 72.24
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10618.00 0.00 336894.40 31.73 0.83 0.08 0.08 81.92
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10827.00 0.00 343294.40 31.71 0.79 0.07 0.07 78.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10549.10 0.00 334728.66 31.73 0.79 0.07 0.07 77.92
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.20 0.00 10795.80 0.00 342539.20 31.73 0.78 0.07 0.07 77.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.60 0.00 9832.00 0.00 311672.00 31.70 0.78 0.07 0.08 77.44
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 3510.20 0.00 533.20 0.00 32654.40 61.24 3.60 10.66 1.80 96.16
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 461.52 0.00 67.13 0.00 4181.16 62.28 3.53 51.02 14.88 99.88
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 2078.49 0.40 4928.88 3.19 166224.70 33.72 44.53 2.56 0.20 97.77
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 184.37 0.00 6050.90 0.00 193478.16 31.98 49.01 13.42 0.17 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 643.31 0.00 20252.89 0.00 647453.89 31.97 48.47 2.39 0.05 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 569.94 0.00 19219.44 0.00 614432.06 31.97 48.60 2.53 0.05 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 484.20 0.00 15684.00 0.00 500665.60 31.92 48.27 2.51 0.06 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.20 0.00 6.40 32.00 43.54 3312.00 5000.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 563.27 0.00 18312.57 0.00 585363.67 31.97 48.18 5.45 0.05 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 615.60 0.00 19248.80 0.00 615324.80 31.97 49.01 2.55 0.05 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 656.29 0.00 21067.86 0.00 673494.61 31.97 48.40 2.30 0.05 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 651.00 0.00 21388.40 0.00 683654.40 31.96 48.52 2.27 0.05 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 676.15 0.00 21180.16 0.00 677162.32 31.97 48.56 2.29 0.05 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 638.40 0.00 21072.60 0.00 673747.20 31.97 48.46 2.30 0.05 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 6479.60 0.00 3066.60 0.00 131396.80 42.85 86.42 19.62 0.30 92.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 298.80 0.00 56.80 0.00 4963.20 87.38 87.00 1606.92 17.61 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 2.80 0.00 7.60 0.00 376.00 49.47 32.10 3630.84 131.58 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 2743.49 0.00 265.73 0.00 21862.93 82.27 29.09 208.17 3.65 96.99
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 236.00 0.00 50.60 0.00 4420.80 87.37 85.77 1609.39 19.76 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.00 0.00 0.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.00 0.00 0.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.99 0.00 0.00 99.84
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.00 0.00 0.00 100.04
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.99 0.00 0.00 99.92
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1497.80 0.20 156.91 1.60 13317.84 84.78 75.93 716.88 6.29 98.76
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1978.64 0.00 184.23 0.00 14987.62 81.35 84.12 391.55 5.27 97.09
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 2081.40 0.00 216.60 0.00 18302.40 84.50 142.87 651.16 4.51 97.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1890.20 0.20 210.60 1.60 17993.60 85.37 108.64 554.66 4.66 98.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1490.00 0.00 155.80 0.00 13212.80 84.81 129.51 839.51 6.31 98.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1430.40 0.00 136.40 0.00 11590.40 84.97 103.09 687.65 7.19 98.08
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1106.40 0.00 110.80 0.00 9492.80 85.68 114.16 1151.88 8.89 98.48
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1937.60 0.00 226.20 0.00 19289.60 85.28 97.60 427.02 4.36 98.56
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 743.31 0.20 73.45 1.60 5202.40 70.66 88.49 986.88 13.42 98.84
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1491.00 0.00 157.40 0.00 13427.20 85.31 88.07 562.37 6.25 98.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1046.40 0.00 96.80 0.00 8260.80 85.34 102.64 990.85 10.15 98.24
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 810.82 0.00 88.78 0.00 7445.29 83.86 130.90 1160.75 11.18 99.24
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1122.95 0.00 112.97 0.00 9708.58 85.94 119.94 1295.33 8.76 99.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 744.40 0.20 87.80 1.60 7441.60 84.58 102.39 1090.86 11.27 99.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 744.51 0.20 96.61 1.60 8183.63 84.55 100.76 1341.66 10.24 99.16
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1497.99 0.00 164.26 0.00 13352.61 81.29 100.77 622.15 6.00 98.63
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 4397.41 0.20 429.74 1.60 36188.42 84.17 88.42 207.36 2.21 95.01
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 3777.64 0.00 426.75 0.00 36083.03 84.55 108.43 260.46 2.24 95.57
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 4892.80 0.20 483.20 1.60 40792.00 84.39 90.34 172.50 1.95 94.16
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 4049.50 0.20 425.45 1.60 36117.03 84.85 106.13 259.25 2.25 95.71
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 3583.60 0.00 373.00 0.00 31508.80 84.47 116.98 305.78 2.56 95.52
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 4599.60 0.20 483.03 1.60 40980.44 84.81 99.17 210.96 1.96 94.53
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 5242.57 0.20 568.88 1.61 48151.00 84.62 85.44 131.88 1.66 94.54
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 348.70 0.00 31.94 0.00 667.47 20.90 101.35 3325.60 30.98 98.92
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 6922.65 0.20 732.26 1.60 61361.12 83.78 130.94 174.46 1.28 93.55
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 6137.33 0.20 650.70 1.60 55326.15 85.00 105.11 171.18 1.42 92.69
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 8535.00 0.40 878.20 3.20 74152.00 84.40 88.46 97.92 1.02 90.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 6773.25 0.20 713.37 1.60 60146.91 84.29 125.94 185.50 1.30 92.77
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 4784.40 1.20 526.20 33.60 44696.00 84.81 48.61 94.11 1.79 94.56
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 2.00 6.80 0.60 217.60 20.80 32.22 1.05 165.30 135.14 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.00 13.40 0.40 428.80 11.20 31.88 1.00 59.07 72.46 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.40 9.40 44.20 40.20 1414.40 396.80 21.46 36.11 429.97 11.85 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.20 135.53 0.40 4336.93 12.77 32.00 1.00 7.34 7.33 99.64
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.60 151.90 0.40 4860.92 16.03 32.02 1.00 6.58 6.57 100.04
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.20 144.40 0.40 4614.40 12.80 31.96 1.00 6.88 6.87 99.52
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.60 1.20 155.40 0.40 4972.80 12.80 32.00 1.00 6.33 6.41 99.92
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.20 52.40 0.40 1676.80 12.80 32.00 1.02 19.48 18.94 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.00 36.47 0.40 1167.13 11.22 31.96 1.01 27.43 27.17 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.80 1.00 38.20 0.80 1222.40 14.40 31.71 1.01 25.85 25.64 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.80 1.20 137.72 0.60 4400.80 14.37 31.92 1.01 7.28 7.19 99.48
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 11.00 2.20 1152.60 0.60 36876.80 22.40 32.00 26.63 20.87 0.86 99.44
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.00 1.20 163.00 0.40 5222.40 12.80 32.04 63.36 389.43 6.12 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.40 1.20 155.60 0.40 4979.20 12.80 32.00 63.12 401.14 6.41 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 3.20 1.20 143.80 0.40 4601.60 12.80 32.00 62.66 436.97 6.93 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.80 1.60 134.40 0.40 4300.80 16.00 32.02 63.00 465.77 7.42 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.80 1.20 159.80 0.40 5113.60 12.80 32.00 63.37 398.41 6.24 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.80 155.00 1.40 4960.00 25.60 31.88 63.39 400.14 6.39 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.20 152.80 0.60 4883.20 14.40 31.93 62.52 408.45 6.52 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 3.20 1.20 153.80 0.40 4921.60 12.80 32.00 62.53 406.60 6.49 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.20 145.80 0.40 4665.60 12.80 32.00 62.59 429.42 6.84 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.80 1.20 138.72 0.40 4439.12 12.77 32.00 61.47 443.82 7.17 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 1.20 1.20 150.80 0.40 4825.60 12.80 32.00 62.25 408.74 6.61 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.40 2.21 103.41 2288.76 3309.24 72536.55 31.71 40.54 18.06 0.39 93.82
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.20 0.00 725.50 0.00 23056.57 31.78 0.98 1.38 1.36 98.33
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.20 0.00 3763.86 0.00 119460.24 31.74 0.91 0.24 0.24 91.08
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.00 0.00 63.87 32.00 1.00 454.40 500.00 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.20 0.00 2.61 0.00 83.37 32.00 1.00 409.23 384.62 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.40 0.00 0.80 0.00 11.18 14.00 5.19 1947.00 1250.00 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.20 0.00 2.00 0.00 52.80 26.40 1.59 2577.20 500.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.60 0.00 2.40 0.00 72.14 30.00 1.00 475.67 416.67 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.80 0.00 3.59 0.00 111.78 31.11 1.00 271.11 277.78 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.00 0.00 3.81 0.00 120.24 31.58 1.00 260.21 263.16 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.80 0.00 2.79 0.00 86.23 30.86 1.00 307.43 357.14 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.00 0.00 2.60 0.00 81.60 31.38 1.00 408.00 384.62 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.80 0.00 3.80 0.00 118.40 31.16 14.13 262.95 262.53 99.76
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 4.39 0.00 140.52 32.00 50.81 4022.00 227.27 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 4.81 0.00 153.91 32.00 34.36 9330.17 208.33 100.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.79 0.00 75.05 26.86 11.43 11746.57 357.14 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 2.01 0.00 3.82 0.00 123.69 32.42 51.05 5032.63 263.16 100.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.99 0.00 95.81 32.00 45.40 7306.40 333.33 99.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.60 0.00 83.20 32.00 31.07 12140.92 384.62 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 1.60 0.00 51.20 32.00 19.51 16378.50 625.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.20 0.00 70.40 32.00 10.86 21505.82 454.55 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 4.60 0.00 1.60 0.00 41.60 26.00 21.67 18743.00 625.00 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.20 0.00 70.40 32.00 49.72 4714.18 454.55 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.20 0.00 70.40 32.00 46.99 9282.91 454.55 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 5.40 0.00 195.20 36.15 46.16 14913.33 185.19 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 3.60 0.00 110.40 30.67 33.73 12728.67 277.78 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 2.80 0.00 89.60 32.00 24.99 11474.00 357.14 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.00 0.00 4.60 0.00 140.80 30.61 11.80 9089.22 217.39 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 8.58 0.00 0.60 0.00 14.37 24.00 2.13 546.67 706.67 42.32
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.40 1.40 3.21 80.16 46.22 1.75 2077.78 153.78 27.74
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 1.60 0.00 0.40 0.00 15.97 40.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

[root@ip-10-251-146-159 ~]#

Is EC2 useful as a database server

Plenty of people have been excited by the prospect of Amazon EC2 and the ability to scale out your databases as load increases from your original configuration. I noticed Morgan Tocker and Carl Mercier are going to be presenting on this topic at the upcoming MySQL Conference

However almost immediately people are worried about the lack of persistent of data across instance terminations.
In a sense people are wanting dedicated hosting services instead of what EC2 really is.

You need to think of Amazon EC2 in a electricity generation metaphor.
Coal, Nuclear Fission and Gas provide the base load electricity which is on 24×7.
Gas and Hydro can act as peak load generation as well. So at peak periods when the requirement for more electricity is higher, electricity generators can switch on these extra resources to cope.

Amazon EC2 is the same, it is there to service peak load.
Either you are using Amazon EC2 as a base load server or you are using a dedicated hosting service to provide base load. You add additional server resources during peak periods as required.
As a dedicated hosting service EC2 is not actually the cheapest option out there. There are plenty of dedicated hosting providers who will give you and your application and database, cheaper base load capacity. That said, many people choose to run both application and database servers on EC2 as base load servers and the uptime of these instances is good.

What this means is that using EC2 as base load means you must implement additional protections for your data to provide persistence. This may be in the form of clustering technologies and replication technologies or both. So running EC2 as a base load database server adds complexity. This is why numerous companies have sprung up as a result of this complexity. They are essentially providing a method for companies to pay for someone else to deal with it.

The hidden value here is, in adopting a more thorough attitude to data persistence and redundancy, your database is more robust. So if or when your dedicated hosting provider has an outage, your architectural design is already in a position to handle it.

The danger is, you see any ongoing performance issue (a demand for addition base load) as solved by throwing hardware at it. Rather than reviewing whether the demand is justified or whether it can be reduced through tuning the application, database or architecture.

Update: Added Carl as co-presenter at the MySQL conference.

Have Fun

Paul