On April 20, the 10gen crew will be paying a visit to the South, putting on MongoDB Atlanta at the Georgia Tech Research Institute (GTRI). In our continuing partnership with 10gen, we are excited to announce our sponsorship of this conference and the events surrounding it.

While MongoDB events are always interesting, there are two presentations that we are especially excited about and wanted to bring to your attention:

MongoDB Aggregation Framework

Obviously, the big news in the coming release of MongoDB 2.2 is the introduction of a powerful, performant, but easy to use aggregation system. Paul Pedersen of 10gen will be presenting on this in the afternoon, covering usage patterns and when it is right to use this framework versus writing your own map reduce functions.

Saving Time with MongoDB

Often, people ask why they should use MongoDB over other database technologies, especially the MySQL standby. Harris Reynolds, a developer with Nimble Labs, is going to highlight how you can quickly make progress on projects with MongoDB, how to create APIs for your data and how, once you start growing, MongoDB can remain performant for you.

It looks to be a great event … if you are in the southeast and want to learn more, you still have time to register. Plus, the GTRI is the perfect place to put on a conference, offering easy access, lots of wifi and plenty of power outlets.

The MongoHQ team be there, so come and say hello!

LessConf 2012 kicked off in Atlanta, Georgia today and this year, MongoHQ had the opportunity to be a sponsor for the event. The goal of LessConf is bringing together entrepreneurial developers and designers … offering them encouragement, “interesting” presentations and opportunities to meet each other.

Steven Bristol and Allan Branch have been friends of ours for a long time and, in a round-about way, had a hand in us taking the leap and starting MongoHQ. So we owe a lot to them, their kindness and their inspiration.

As founders ourselves, we love finding creative ways to give back to the developer community and LessConf is just one of the avenues. If you have a hackathon, a community project, developer meetup, whatever … let us know and maybe we can help. All great things start off with small starts … where a couple of people were given a chance by someone to make something happen. The world gets better that way.

The team at MongoHQ is proud to announce a partnership with 10gen, the creators of MongoDB. This partnership will help expand MongoHQ’s professional support and expertise by providing critical access to 10gen’s 24/7 support and engineering teams. This will also allow us to provide deeper support offerings, and handle even larger and more complex MongoDB setups.

What this means for you as a user of MongoHQ, is even more peace-of-mind. You are not only working with the best MongoDB hosting team available, but you will have the experts that created MongoDB backing them up as a third tier. Whether you have huge custom installations, or smaller shared databases, you will benefit from even greater uptime and reliability.

Our focus is offering the best customer experience possible and by expanding the capacity of our relationship with 10gen, this is a big win for our customers. We have come a long way since the pre-1.0 days of MongoDB and the product and community that now exists has made it all worthwhile.

• More About MongoHQ / 10gen Cloud Hosting
• 10gen’s Partnership Announcement

More details on this new arrangement will be announced soon.

In the first of many posts to simplify connecting to MongoHQ, we’re providing sample code for many programming languages, frameworks and setups (replica sets, slaveOk, etc).

These examples all assume that your MongoHQ connection string is set in an environment variable MONGOHQ_URL, like this:

export MONGOHQ_URL=mongodb://user:pass@server.mongohq.com/db_name

If you use any of our PaaS providers this environment variable should already be set when you choose the MongoHQ add-on. Or you can set it yourself using whatever mechanism the Paas allows. For example, in Heroku:

heroku config:add MONGOHQ_URL=mongodb://...

Today I’ll roll out PHP, two NodeJS (JS and Coffee), and Ruby snippets. In every case we use the basic driver to connect (no frameworks like CakePHP or Mongoid–those will come later).

PHP Connection to MongoHQ

This originally began as a Gist from Larry Hitchon of AppFog. I fixed it up a bit and made it MongoHQ-centric. It’s a pretty straightforward driver. If you don’t have PECL, you should install PECL.

sudo pecl install mongo

NodeJS Connection to MongoHQ (JavaScript)

This code uses the node-mongodb-native driver, though in production you may want something a little less… nesty. Like all good Node packages, you can get it via NPM.

npm install mongodb

NodeJS Connection to MongoHQ (CoffeeScript)

(Side note… I love coffeescript)

Ruby Connection to MongoHQ

We wrap up today by connecting Ruby to MongoHQ. To get started, install the mongo gem. If you want more speed, install the native bson_ext, and if you are in less than Ruby 1.9, you’ll need the json gem. It should go without saying, but you will need RubyGems.

gem install mongo bson_ext json

Part 2 will be Python and Java, and maybe more.

I planned to write a long, drawn out post about MongoHQ’s new webapp design that we’re rolling out to select customers today. I planned to entice you with juicy words. But since a picture is worth a thousand words, here are 4000 words about our new chart application:

Choosing a Database

Following Database Operations

Lots of Little Details

Works in Mobile

A good design keeps end-users in mind to craft a first-rate user experience (UX): straightforward workflow, clutter reduction, beautiful and joyful to use, and thousands other good rules of thumb. A good API follows many of the same rules. But rather than buttons and graphs, it manifests itself as beautiful markup and unambiguous actions. APIs are UX for programmers.

We are in the midst of an overhaul, of sorts, to much of our internal and external codebase. We’ve replaced the gerbils who used to perform all of the MongoHQ work with an API (which wraps even smarter gerbils, but we’ll focus on the API for now). We wanted to ensure that we designed something we wanted to use, so these new systems all interacts through our new API. We are eating our own breakfast here, to make sure it’s tasty and healthful before serving you.

The new API isn’t ready for prime-time, yet. But it’s out there, and you’re welcome to take a look. This is still a VERY BETA product, so bear in mind we make changes and improvements all the time.

API Example

Unsolicited Example

Before digging into the API, let’s see what you can do with the MongoHQ REST API. David Knell created this cool MongoHQ client module for Appcelerator (mobiles devices). He doesn’t work for us, and we didn’t solicit him to do it. Check out his video:

My favorite part is when he creates a database from our list of plans. One click create-deploy-use! This is exactly the kind of open ecosystem we’re trying to build by making MongoHQ the most transparent and fully featured database host, while holding to our ideals of simplicty and reliability.

Getting a Key

The API is simple to use. But you’ll first need a MongoHQ account (sadly, a database created through a PaaS like Heroku or AppFog won’t work–you’ll need to sign up). Log into the web application, and click on “My Account” in the upper right. There you’ll find your secret API key, also known as _apikey.

Creating a Database

You can find out what plans we offer by issuing a GET request to the /plans path. All api requests hit the https://api.mongohq.com endpoint. We’ll use curl for convenience, but you can use anything that issues an HTTP GET request. Just be sure to replace XXXXXXX with your own private API key (also, don’t let it fall into the wrong hands!).

curl -X GET "https://api.mongohq.com/plans?_apikey=XXXXXXX"

Once you’ve picked the plan you want, issue a POST to /databases to create a new one. You’ll need two JSON values: a name to give this database, and the type of plan you want (we’ll just pick free).

curl -X POST "https://api.mongohq.com/databases?_apikey=XXXXXXX" \
-H 'Content-Type: application/json' \
-d '{"name" : "test_db"}'

You can make up to 5 free databases this way in a single account–of course there’s no limit to how many larger plans you want to make.

Great Power

This was just a quick demo for making a new database through the MongoHQ API. If you read the docs, you can see how to create/query collections/documents/indexes, check invoices, view stats (for larger plans), and more.

As a reminder, this is a beta API, so keep that in mind before you bet the farm on it. Feel free to play around, and by all means, make suggestions. We love your feedback.

MongoHQ is preparing to span the globe and, like all domination plans, it begins with Europe.

We have for quite some time dedicated plans in Europe on a limited basis. MongoHQ has just launched replica set plans to the European West AWS datacenter, by adding an eu-west-1 (Ireland) option. If this is successful, we aim to offer this datacenter option to our other shared plans. (Of course, we’ll host custom deployments anywhere in the world you wish–so there’s never a need to wait if your data needs are big enough.)

If you have concerns about a US company hosting European data–we have recently become EU and Swiss Safeharbor certified self-complient with the US Department of Commerce. In short, it is a certified commitment to adhere to European laws of data privacy as outlined by European, Swiss, and United States treaties.

Finally, MongoHQ is going on a quick European tour (of sorts) in late April. The current plans are meeting the London MongoDB Group on April 17, Railsberry in Krakow Poland April 19-20, and leaving the EU on April 23. Come meet up in London or Poland (or maybe a third location? We have nothing planned on 21-22, and have been known to travel great distances for beer).

After months of coordination and rumors, today I’m happy to announce that MongoHQ is a supported add-on for AppFog (you may be familiar with their first offering: PHPFog).

MongoHQ was founded on the idea that data management should be easy—so developers can get down to the business of coding rather than spending time on complex database operations. We believe that simplicity and robust scalability can co-exist. Like MongoHQ, AppFog shares this belief, which is why we are so excited to begin this partnership. AppFog and MongoHQ customers will benefit greatly as our integration makes setting up and managing a scalable cloud hosted application that much easier.

This is MongoHQ’s 5th PaaS partner, along with Heroku, AppHarbor, Nodejitsu, and CloudBees—and the first focused largely on PHP (though they are growing beyond that).

Replica sets are cool. They add data redundancy in a mostly transparent way, requiring only a little extra developer knowledge. That knowledge is important, though, and we regularly help our customers troubleshoot applications running into preventable problems.

Replica sets in one tweet or less

Replica sets are master/slave clusters that automatically “promote” a slave if the master DB becomes inaccessible.

Or read a longer description.

What a developer needs to understand

MongoDB handles the intricacies of syncing data, managing which member is primary, and recovering secondaries for you. It doesn’t, however, make your client code any smarter about which member to talk to (well, unless you’re running mongos, but that’s less common). Applications themselves should know about as many replica set members as possible and be able to switch between them on the fly.

The most basic requirement for a replica set is that all writes must go to the primary set member. Secondaries can handle reads, but will complain if they encounter a write command. Most drivers will figure out which member to write to, though… assuming they can connect to any member. This brings us to our first rule of speaking replica set:

Rule #1: Always provide your driver with as many replica set members as possible

Many applications only connect to a single host on startup. This is bad, and not for any good reason. if that one host isn’t available at connection time, the application will behave as if the entire replica set is down. Drivers options differ slightly, but they all have options for connecting to multiple replica set members. If you tell them everything you can about your setup, they will usually do the right thing when connecting.

Rule #2: Handle connection failure exceptions

Secondaries are most commonly promoted when the primary becomes completely inaccessible. This can happen for a number of reasons, anything from a routine mongod restart to a helicopter “landing” in the middle of your datacenter. Drivers will generally raise a failed connection exception when they send an op to a missing replica set member, then leave the rest up to you. You can choose to either retry the operation (a good idea for critical writes) or just give up and wish your users better luck next time (a good idea if you don’t like your users).

Rule #3: Be ready for “not master” exceptions

Drivers won’t always raise connection exceptions when a primary “steps down” to secondary. We usually see this happen when we demote the primary member of a replica set for maintenance purposes and an application keeps on sending writes along. MongoDB will send errors back indicating that it’s “not master”, but applications may not know what’s going on.

This is a little more tricky to deal with, unfortunately. If an application is writing in safe mode, drivers will usually raise an exception with “not master” in the message string. If an application isn’t using safe mode, and never checks the getLastError command it will happily keep on sending data into oblivion. In a complete coincidence, rule #4 takes care of this problem:

Rule #4: Practice safe writes (unless you can afford to lose them)

Simply using safe mode will help you catch failing updates, but there’s a bit more safety to be had in a replica set setup.

In single server MongoDB setups, safe mode ensures that the write “succeeded”, either in memory, on disk, or to the journal (depending on the options). Replica sets get to use another flag that tells the command to hang out and wait until the write has been replicated to other replica set members. If your data is important, you probably want to verify that writes are committed to a majority of replica set members (the “majority” flag).

There’s no flag for “chisel this into granite and store it in a converted underground missile silo for future generations to unearth after the inevitable llama uprising”, unfortunately. 10gen does take feature reqeusts.

Rule #5: Go ahead and read from secondaries

You may as well spread queries across your secondaries. They’re basically just sitting there with nothing to do but keep up with writes. You could even be extra clever and have a multi-datacenter replica set (we’ll set it up!) for disaster recovery purposes, host your app in multiple places, and read from the closest replica set member.

As an aside, secondary replica set members are a great way to get around the one-at-a-time map/reduce limitation in MongoDB. We have a number of customers who run “map/reduce secondaries” that will never take over as master, but run map/reduce jobs all day.

Don’t waste your replica set

It’s sad to have a nice, shiny replica set setup and an application that doesn’t understand how the data store works. In the worst cases, you’ll lose data despite a fully functional database. On normal days, your app may benefit from spreading reads around to otherwise bored servers. Make sure you teach your applications to speak “replica set”.

All of our free/sandbox databases were successfully upgraded to MongoDB 2.0.2 a couple of days ago. The upgrade was very smooth and our large community of users have access to the latest release of MongoDB.

So, Why the Wait?

Over the last few weeks, we tested the various MongoDB 2.0 releases (from 2.0 to 2.0.1 to 2.0.2) and saw the need to work directly with 10gen to incorporate a couple of important fixes. These were aimed at preventing a sizable number of MongoDB installs from having driver issues related to how we run our hosted databases in authenticated mode.

Once these were in place, we were ready to move forward.

Reliability is extremely important to us, so before doing large-scale upgrades, we wanted to make sure that the updates would be seamless.

What’s Next?

Now that we have successfully upgraded our sandbox environments, we are preparing announcements for upgrading all of our environments to MongoDB 2.0.2. Also, we will be adding some follow-up articles that talk in more detail about the benefits of MongoDB 2.0 and how you can take advantage of them.

Support

If you have questions about the upgrade or need assistance with driver support, please contact us at support@mongohq.com. We are always happy to help!