MySQL 5.1 NDB Cluster on EC2 – Part 2

Posted by

9

Given I had already been through the procedure of setting up an MySQL 5.0 NDB Cluster on EC2, getting the MySQL 5.1 NDB Cluster installed was a reasonably straight forward task.

I will create a new HOWTO wiki from my screen dumps in the next day or so though the documentation provided by MySQL is very thorough.

Given I want to give a large MySQL 5.1 cluster a whirl, maybe with Cluster replication, as described in this documentation, I built and bundled the following AMIs (Amazon Machine Images)

  1. MySQL 5.1 NDB data node – built from the rpms
  2. MySQL 5.1 NDB Management node – built from rpms
  3. MySQL 5.1 NDB SQL/API node – built from rpms
  4. Complete MySQL 5.1 install.

I built number 4 as you can run the whole cluster to test everything off one box or in my case image. You could also run the management node and SQL node on the same box as well.

The biggest issue I found which is specific to running on EC2 is the hostname, which is allocated via DHCP. I am working on scripts to automate the updating of the /etc/hosts file on each box so that the configure.ini required on the management node and the /etc/my.cnf settings required on the data nodes can point at a name rather than a specific IP address or DNS name.

The work I do here can be replicated to handle the hostname stuff required for Openfiler, Oracle standbys and anything else requiring network connectivity.

For particular interest for users of EC2 is the lack of persistent storage on any specific instance. Plenty of people are looking at various solutions, I guess the interest in using MySQL cluster is that any one data node or two (if you are paranoid) could die so running enough data nodes specifically and also maybe sql nodes with management nodes on the same would provide a solution to this. Backing up to S3 or some form of persistent storage is the fail back if everything goes pear-shaped.

Replicating these virtual images to boxes with persistent storage alleviates some of the pain, however EC2 remains a great area to built your knowledge of various technologies available. Even if some of quirks of running under a virtual machine make things slightly different.

If there is interest I will release a public image of the various types of nodes.

Have Fun

Paul

Part 1: Why use MySQL 5.1 NDB Cluster?

9 thoughts on “MySQL 5.1 NDB Cluster on EC2 – Part 2”

  1. So, basically you put MySql Cluster on 4 separate EC2 instances? Can you make the 4 AMIs shareable? The way i have been thinking of utilizing EC2 is – Have 1 or 2 app servers, 1 hardware level load-balanced webserver and the complete MySql cluster, all with an external hosting company like Layeredtech. After that, keep adding additional webservers and app servers on EC2 depending on the load during the day.Somehow i don’t feel confident about putting production level database on EC2 given the fact that you cannot really reach a tech support guy if you really need to.

  2. Having the MySQL cluster on 4 AMIs is probably sufficient for redundancy especially if you increase the number of management nodes and SQL nodes as well.Most of the MySQL documentation talks about having dedicated nodes for each part of the cluster, however you can run multiple parts on one node.I will at some stage make AMIs sharable, I am guessing that if I don’t someone else will given the ease at installing it.Using EC2 as a overflow resource for app and webservers is a good idea. I would treat having access to a tech support guy similar to how you treat redundancy. If you ran enough machines for redundancy is there any need?If you are talking performance related issues that is different and there plenty of companies who provide those services.

  3. Hi!I would be interested in getting access to your AMI’s.Actually i’m really interested in a follow up to this.I think it is a fascinating proposition to get some sort of database cluster running in a amazon group.. along with Web.How did you work out the DNS/DHCP issues?

  4. Hmmm I did a quick scan of the available AMIs and noone has made a MySQL NDB cluster set available.Which is surprising.DHCP/DNS is a pain, I have written some scripts to grab the hostname and update the configuration files.I will post a new comment to cover my thoughts on the whole database on EC2 issue soon.

  6. I have a couple of outstanding things before I return to the MySQL NDB cluster.Once I have a 64 bit MySQL NDB AMI. I will release as a public AMI.Given that the MySQL NDB (at least for 5.0) is an in-memory db using the 32 bit instance is not going to give you much db space.Given this post has been popular, once the sysbench and mysql proxy stuff is out of the way. It is back to Cluster land.

  9. I will answer the two outstanding questions.

    @Ed:
    I did a quick search of the Amazon AMI, didn't find a 64 bit AMI for MySQL Cluster.
    I will followup with getting an AMI built. I might setup a survey to see what demand there is for this AMI.

    @Alex:
    I searched and found some articles on running WordPress on MySQL Cluster. So it can be done, I didn't find specific solutions for running WordPress on EC2.

    There are two problems with running WordPress on EC2:

    1) EC2 can be used as continuous loading (base load) but is really priced for peak load to use the electricity analogy. So running off EC2 will be expensive vs other VPS or dedicated hosting.

    2) You can NOT add more nodes to MySQL Cluster without restarting the whole cluster. If you wanted to offload traffic to EC2 in a peak situation, the better solution would be standard MySQL replication. WordPress has a HypeDB plugin to handle this specific scaling option.

Comments are closed.