By triggering real-time time alerts, it will bring a better understanding of growth and scaling in staging and early production.

Get alerts on the Go

With this feature, we enabled multiple methods of message delivery. You can

• View alerts on the website via the “Alert” icon
• Receive an E-mail when an alert is triggered
• Receive a PagerDuty notification when an alert is triggered

By default, all alerts are viewable via app.mongohq.com. To have alerts delivered, click the ‘Alerts’ icon and choose “Settings”. All delivered alerts will be metered to 1 unique alert every 12 hours.

When you get an alert on the go, remember, MongoHQ’s site is built to be responsive from mobile devices. Get the alert, and handle the alert immediately while on the go.

Indexing Constraints

The following constraints are in effect as of MongoDB 2.4.x release. There is talk of adding features in later versions that will remove most of these limitations. If these limitations are removed, these constraints will continue to be the best practice for scaling, and will lead to better performance.

• MongoDB can only use 1 index per query
• MongoDB can only use one “multi-value” operator / query (e.g. $nin, $in, $nor, $gte, $ge, $lt, $lte, $near, $sort). Yes, that does include sorting. However, you can use a range query and sort on the same field effectively.
• MongoDB indexes must include the multi-value operator as the last used field in the index.
• RAM is fast; disk is slow. Have enough RAM for your MongoDB indexes.

Below are example queries, one that breaks the constraints above, and one that shows a solution:

1
2
3
4
5

// A Bad query with conflicting `time` and `user_id`
db.coll.find({action: "run-process", time: {$gte: ISODate('2013-05-06'), $lt: ISODate('2013-05-07')}).sort({user_id: -1})

// A better query using date 'bucketing'
db.coll.find({action: "run-process", day_bucket: ISODate('2013-05-06')}).sort({user_id: -1})

When optimizing queries, move logic from the query into the schema. Instead of using the query to bound a date field, we are using a date bucket to move that date logic to the schema. In the example above, we remove the $gte and $lte, and replace with a bucket value representing the entire day.

These indexing constraints will set you free. By following these constraints, you will model data and build a database that can be scaled in a sharded/distributed environment. These constraints require you to be creative with your schemas.

Too Many Indexes

Everyone thinks of the new indexes to add. Not everyone thinks of indexes to be removed.

Recently, I helped a customer optimize his database. Write lock on the database was running consistently at 95%. CPU was spiking consistently, and making for a poor experience. We looked at the indexes, and determined the customer had too many indexes (126 non-_id indexes). 24 of the indexes were on one collection. For each insert/update which modified keys, MongoDB was writing the document, and updating 24 indexes. This caused of the excessive write-lock.

The first action was to delete all of his indexes (do not try this on a database larger than 15 GB). Immediately after deleting the indexes, write lock fell to 0%. A tradeoff for removing indexes was going to disk for queries. However, disk reads were faster with no indexes than when performance writes against too many indexes.

Once we removed these indexes, we turned on system profiling in MongoHQ from the Database > Admin page. We let the database run for about 6 hours, and then ran a short analysis on the system.profile. In the analysis, we found 6 essential indexes. After adding the 6 essential indexes, performance was drastically better. In summary, we removed 124 indexes, and added 6 back as required.

For the sake of RAM and write lock , be precise with your indexes.

Best Practices

• Know thy constraints above
• Learn to use explain explain.explain() - Understanding mongo query behavior
• Limit the number of indexes on a collection. The first index will not hurt performance. The 24th index will hurt performance. It is up to you to know what is acceptable. Keep track of your indexes and queries.
• Keep track of your indexes and queries. MongoDB does not have an index usage counter. If you think one would be valuable, go here to vote! https://jira.mongodb.org/browse/SERVER-2227

The MongoHQ Team is in Atlanta today, sponsoring and presenting at MongoDB Atlanta. This conference, in its third year at at Georgia Tech Research Institute (GTRI) has a host of great speakers, including our own Chris Winslett. I’m sure the southern food and hospitality will be enjoyed as well. Check out some of the great talks that are happening today:

MongoDB Index Constraints and Creative Schemas

The first step to understanding scaling with MongoDB is understanding the constraints of the system. We will build out the constraints of MongoDB indexing. Then, we will talk about how to use these constraints to optimize schema. Presented by: Chris Winslett, MongoHQ.

Developer Happiness Through MongoDB

MongoDB is the ideal data store for modern web apps. By providing freedom of choice, a sense of creative control, and a strong adaptability to change, MongoDB makes developers happier. Problems that are tedious or unnecessarily complex to model with a relational database are instead a joy to solve with a schema-less, documented-oriented structure. Those factors along with a suite of built-in tools like full-text search and aggregations contribute to an excellent experience for web developers. Presented by: Luigi Montanez, Upworthy.

Hash-based Sharding in MongoDB 2.4

In version 2.4, MongoDB introduces hash-based sharding, a new option for distributing data in sharded collections. Hash-based sharding and range-based sharding present different advantages for MongoDB users deploying large scale systems. In this talk, we’ll provide an overview of this new feature and discuss when to use hash-based sharding or range-based sharding. Presented by: Kelly Stirman, Director of Product Marketing, 10gen.

The team at MongoHQ is excited to be a part of this great event and we are looking forward to particiaption in the upcoming MongoSF and MongoNYC events as well.

1
2
3
4
5
6

db.events.aggregate(
  {$match: {time_bucket: "2013-01-month"}},
  {$unwind: "$time_bucket"},
  {$project: {time_bucket_event: {$concat: ["$time_bucket", "/", "$event"]}}},
  {$group: {_id: "$time_bucket_event", event_count: {"$sum": 1}}}
)

Let’s take a step back and look at the different options: Aggregation Framework or Map Reduce?

Aggregation Framework or Map Reduce?

Since 2.2.x release, the aggregation framework has been the de facto MongoDB number cruncher. If you are familiar with a SQL db’s group by functions, you will be at home with the functions. Performance wise, Aggregation Framework smokes MapReduce, like not even close.

Unless your required data manipulation functions are not present in the current versions, chose Aggregation Framework. For instance, different types of statistical analysis may be difficult with the Aggregation Framework. Averages are easy. Any type of deviation calculations will require aggregation process calls: one for averaging and the other for deviation calculations.

In sharded MongoDB environments with aggregation framework, you will get the benefit of distributed processing at the data node level as you did with map reduces. Thus, each data node will return the summated results, and the mongos will concatenate and process the results returned from data nodes.

Getting started with Aggregation Framework

Aggregation framework is a series of actions, also known as a “pipeline”. This pipeline is processed in order and each action can filter or manipulate data. For instance, if you wanted to filter data and concat a variable called name:

1
2
3
4

db.my_collection.aggregate([
  {$match: {first_name: "Clark"}},
  {$project: {_id: "$id", full_name: {$concat: [$first_name, " ", $last_name]}}}
])

The above will return a full name for all individuals with first_name “Clark”. For a complete list of the functions with the aggregation framework, see the 10gen Aggregation Framework Reference.

Processing and Caching

Repeat after me: cache aggregation queries. While running aggregation queries is fast, it is best not to run these commands for every application action. As with most activities, running this command once is fast. Running this command 1000s of times per second will deteriorate performance in the rest of the stack. Also, these analytics systems form the backbone of big data; it is small now, but it production these systems grow quickly.

So, we will run these aggregation commands once, and cache the results. For best results on application performance, try running these actions asynchronously. Trigger the Aggregation Framework through background worker that will run and save the updated values to your aggregation collection.

You will need to run two commands to save the output: 1) the aggregation command and 2) the upsert command. Upsert is a good use case here to update, or insert if does not exist.

Measurement and Summary

After storing data in a best practice way, summating and caching data is the next step for any analytics platform. There is a right way and many wrong ways, and simple algorithm changes can yield good performance gains. While you are measuring your ability to run these analytics systems, you should also measure the performance of these summation queries. Build the measurement of the queries while you are building the query – perhaps in the same background job.

I can guarantee your summation jobs will behave differently with 50 GB of data versus 200 MB of data in a development environment. If you do not measure these summation jobs, you will wake up with performance that you have no historical record. Knowing how you got to that point is unknownable without metrics from the beginning. Furthermore, knowing what is “good” is also a shot in the dark. Measure. Measure. Measure.

Go forth, take your data, and summate.

Why the forced upgrade?

This emergency upgrade was required because of a vulnerability affecting older versions of MongoDB (MongoDB 2.0 and MongoDB 2.2). The vulnerability (SERVER-9124) was caused by deficiencies with javascript sandboxing, allowing the possibility of system commands to be run and if properly exploited, could have resulted in access to other data on the host.

Working with 10gen and understanding the core issue, we made the decision to move quickly to upgrade all of our shared database plans, starting with large databases and finishing, today, with our sandbox/free plans.

Affected environments

The vast majority of our hosts (including all custom deployments) are not at risk from this vulnerability. We’ve isolated Mongo processes on most of our shared hosts in such a way that user data is isolated and individual database vulnerabilities are contained. Some of our older shared hosts haven’t yet been migrated to the new containers, we’re now speeding up our efforts to replace those. Sandbox database hosts are, by definition, going to be more susceptible to problems like this since multiple users share a single Mongo process, we will continue to aggressively upgrade Mongo versions on these.

Email notification

We received reports of some customers not receiving notifications of the upgrade. We are very sorry about this and have resolved the issue. We strive to provide an excellent customer experience and proper communication is an absolute must.

Thank you again for your patience. We have talked to many of you today as we worked through this upgrade together, but if you are having any additional issues or have questions, we’d love to help. You can reach our team at: support@mongohq.com.

We have updated MongoHQ to include MongoDB 2.4.1 and you can safely update your replica sets. If you are using 2.4.0 and have a single/standalone instance of MongoDB, this release does not affect you.

To upgrade, visit the MongoHQ version selector on MongoHQ.com or sign up for a MongoHQ account and try out MongoDB 2.4.1.

If you are interested in finding out more information about this issue, view SERVER-9087 on the MongoDB Jira site.

Along with this, at MongoHQ we are rolling out a new feature that allows you to choose the version of MongoDB you use, simply by selecting a dropdown in the MongoHQ web interface.

This is how it works:

1. Log into MongoHQ and select the database you’d like to upgrade. (If you don’t have an account, you can create one at MongoHQ.com)

2. Click on the “Admin” tab for the selected database, scroll down and choose the version of MongoDB that you want to run.

   

3. Click the “Change Version” button.

That’s it. Now, the magic happens. Ok, just kidding … but clicking the “Change Version” button will initiate a job to change the version of your MongoDB database. This is what happens:

Upgrading Standalone/Single instances

For people using our shared single/standalone MongoDB instances, the update process is simple. Once initiated, we will update the version of MongoDB and perform a brief restart of your database. It should take, at most, 5-10 seconds to complete. From there, you can resume normal database operations.

Upgrading Replica Sets

Upgrading replica sets are a bit different, so the system will perform what we call a “rolling upgrade”. This allows you to migrate to a different version of MongoDB with little to no downtime.

The rolling update will first update all the secondary members and arbiters of the cluster one-by-one, changing the version, restarting and safely bringing the secondary processes back online. After all of this has happened and has verified successful, the system will trigger a state change of your primary to secondary and will allow MongoDB to promote a new primary.

Once the new primary has taken over in the cluster, the system will upgrade the old primary (now secondary) to the version of MongoDB you selected. That will complete the upgrade process.

Please Note: For replica sets, ensure that your driver is properly configured to handle MongoDB replica set state changes, otherwise you may have to restart your application. If you have questions about this, we can help! Just send us a note at: support@mongohq.com.

Supporting our Providers

Of course, if you are running MongoDB with MongoHQ on our providers, like: Heroku, CloudBees, AppFog, AppHarbor, Engine Yard, Nodejitsu and others, you can use this same version selecting functionality to run MongoDB 2.4 as well.

MongoDB Full-Text Search

Since full-text search is considered beta for the 2.4 release of MongoDB, we do not enable it by default when you upgrade to MongoDB 2.4. If you would like to try it on larger data sets, please contact us at: support@mongohq.com and we will work with you to make this happen.

We hope you enjoy this great new feature and that it gives you the flexibilty and control to manage the versions and features of MongoDB that you need to use.

Fixed Length Array in MongoDB

With MongoDB 2.4, one of the cool features is atomic fixed length arrays. To use, combine the $push and $slice functionality. Previously, to have fixed length arrays, you would need to run two separate commands: $find and inplace $set of the entire array.

Now, you can create fixed length arrays with the {$push: {"field_name": {$each: ["array_of_elements"], $slice: -4}}} where field_name is the attribute on the document, and slice is the length of the document (always negative or 0). Essentially, the example above will:

• Push to the right side of the array
• Keep the 4 elements on the right side of the array

An Example

We are going to record the last 5 readings for a river gauge:

1
2
3
4
5
6
7
8
9
10
11
12
13

> doc = db.gauges.insert({_id: ObjectId("513cc4c663c0a2303e1ec093"), name: "Niagra River"})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {lastest_gauge_readings: {$each: [{cubic_meter_second: 5796, at: new Date()}], $slice: -5}}})
> db.gauges.find()[0]
{
  "_id" : ObjectId("513cc4c663c0a2303e1ec093"),
  "lastest_gauge_readings" : [
    {
      "cubic_meter_second" : 5796,
      "at" : ISODate("2013-03-10T17:46:00.810Z")
    }
  ],
  "name" : "Niagra River"
}

As you can see the $push syntax has changed. It is using a more verbose, more functional syntax. In the example above, with $push, we are adding each element in the $each array to the right side of the index. Then, with $slice, we are keeping the 5 right most elements. Below, we will run the following 5 more times with random values:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {lastest_gauge_readings: {$each: [{cubic_meter_second: 5800, at: new Date()}], $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {lastest_gauge_readings: {$each: [{cubic_meter_second: 5960, at: new Date()}], $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {lastest_gauge_readings: {$each: [{cubic_meter_second: 6000, at: new Date()}], $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {lastest_gauge_readings: {$each: [{cubic_meter_second: 6200, at: new Date()}], $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {lastest_gauge_readings: {$each: [{cubic_meter_second: 6300, at: new Date()}], $slice: -5}}})
> db.gauges.find()[0]
{
  "_id" : ObjectId("513cc4c663c0a2303e1ec093"),
  "lastest_gauge_readings" : [
    {
      "cubic_meter_second" : 5800,
      "at" : ISODate("2013-03-10T17:48:31.508Z")
    },
    {
      "cubic_meter_second" : 5960,
      "at" : ISODate("2013-03-10T17:48:37.268Z")
    },
    {
      "cubic_meter_second" : 6000,
      "at" : ISODate("2013-03-10T17:48:42.308Z")
    },
    {
      "cubic_meter_second" : 6200,
      "at" : ISODate("2013-03-10T17:48:45.917Z")
    },
    {
      "cubic_meter_second" : 6300,
      "at" : ISODate("2013-03-10T17:48:49.247Z")
    }
  ],
  "name" : "Niagra River"
}

The $slice operator keeps n-right most elements of the array.

Sorted Fixed Length Arrays in MongoDB

If you like Redis’s zscore functionality, and want to bring it into MongoDB. The latest MongoDB 2.4 is bringing the sorted arrays functionality to MongoDB with the added $push features.

Common scenarios for this are like:

• Content management system - related posts, top n refererring products
• E-commerce storefront - top n referring products
• Social Aggregation - top n retweeted status updates

Sorted arrays are a nice way to cache values instead of using a sort on another query. Try to use these instead of sort queries against entire collections. We will continue to use our river gauge example on the Niagra River:

1

> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {highest_gauge_readings: {$each: [{cubic_meter_second: 5796, at: new Date()}], $sort: {cubic_meter_second: 1}, $slice: -5}}})

Above, we are using $sort to specify a key(s) to sort from, then using $slice to keep the last 5. We will run 5 more pushes to the document:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {highest_gauge_readings: {$each: [{cubic_meter_second: 5796, at: new Date()}], $sort: {cubic_meter_second: 1}, $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {highest_gauge_readings: {$each: [{cubic_meter_second: 5900, at: new Date()}], $sort: {cubic_meter_second: 1}, $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {highest_gauge_readings: {$each: [{cubic_meter_second: 5600, at: new Date()}], $sort: {cubic_meter_second: 1}, $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {highest_gauge_readings: {$each: [{cubic_meter_second: 2500, at: new Date()}], $sort: {cubic_meter_second: 1}, $slice: -5}}})
> db.gauges.update({ _id: ObjectId("513cc4c663c0a2303e1ec093")}, {$push: {highest_gauge_readings: {$each: [{cubic_meter_second: 7500, at: new Date()}], $sort: {cubic_meter_second: 1}, $slice: -5}}})
> db.gauges.find()[0]
{
  "_id" : ObjectId("513cc4c663c0a2303e1ec093"),
  "highest_gauge_readings" : [
    {
      "cubic_meter_second" : 2500,
      "at" : ISODate("2013-03-10T17:59:59.315Z")
    },
    {
      "cubic_meter_second" : 5600,
      "at" : ISODate("2013-03-10T17:59:51.695Z")
    },
    {
      "cubic_meter_second" : 5796,
      "at" : ISODate("2013-03-10T17:59:11.096Z")
    },
    {
      "cubic_meter_second" : 5900,
      "at" : ISODate("2013-03-10T17:59:46.215Z")
    },
    {
      "cubic_meter_second" : 7500,
      "at" : ISODate("2013-03-10T18:00:05.345Z")
    }
  ],
  "name" : "Niagra River"
}

Other Features

If you are looking for a comprehensive list of MongoDB features, take a look at MongoDB 2.4 Release Notes. To begin using a Sandbox database with MongoHQ, Sign Up and create a Sandbox database. To upgrade any paid databases to 2.4.0-rc2, please send an email to support@mongohq.com with your database name.

What we had

Before CircleCI, we used a self-hosted Jenkins instance. Jenkins works well. Organizing jenkins to deploy were running two independent projects: one for testing code, and one for packaging and publishing the resulting asset.

This system only built assets for the master branch. All commands for testing, packaging, publishing were stored directly in Jenkins instead of being under version control.

Our setup was ok, but life can be easier. But, we found the time spent nursing our Jenkins environment to be painful.

Goals

• testing, packaging, publishing steps SHOULD be stored in code
• sensitive credentials should NOT be stored in code
• ci should simply call test, package, publish steps
• ci should publish assets for master, stage, and dev branches

Testing, packaging, and publishing steps

Being fans of #badassrockstartech, we are using Grunt, which is a modular Make system. We use Grunt to run our testing, packaging, and publishing steps.

Powered by npm, we can utilize 3rd-party code to accomplish common tasks (e.g. transpiling coffeescript into javascript) and can define our own tasks (e.g. packaging and publishing to s3).

By moving the logic of testing, packaging, and publishing out of Jenkins and into Grunt, configuring CircleCI becomes very simple and intuitive.

Don’t commit credentials

We moved our commands to publish to S3 into Grunt, but did not want to commmit our credentials. This is one thing we want to remain hard-coded in our CI settings separate from our code.

To accomplish this, I decided a simple export AWS_SECRET_KEY=1234 as a command hard-coded into CircleCI’s web interface would do the trick, but due to the way CircleCI executes your commands you cannot alter the environmental variables in this way.

I reached out to Circle for help, and David Lowe at Circle responded within a few hours with a solution.

We should be adding a secure credential storage mechanism soon, but for now, you will need to work around it.

You’ve got the right idea, but unfortunately, ‘export AWS_SECRET_KEY’ won’t quite work because each command runs in a separate shell. Instead, if you want to do this from the UI, you need to do something like:

echo 'export AWS_SECRET_KEY=1234' >> ~/.circlerc

The .circlerc file is sourced before each test and deploy command.

This works perfectly for our process. We hard-code our sensitive data bits into the pre-dependency section of the web UI.

Simply call test, package, and publish steps

Now that our tasks are defined and our build server configured with credentials, the only thing left is to do all the CI work.

The web UI is an easy way to specify commands to run, but you get more options and flexibility in defining a circle.yml file to tell CircleCI what to do.

This is our circle.yml file.

1
2
3
4
5
6
7
8
9
10
11

## Tests
test:
  override:
    - grunt test

## Deployment options
deployment:
  default:
    branch: [master, stage, dev]
    commands:
      - grunt clean coffee package publish:"$CIRCLE_BRANCH"

Conclusion

Being a service company ourselves, we value offloading tasks to others so we can focus more on what we’re good at. CircleCI integrates seamlessly with GitHub, allows us to customize what we need, and gives us a very pretty an intuitive UI with very little effort.

I’m happy to hear they just raised a bunch of money, and I’m excited to see how they continue to make the world of continuous integration easier.

SSDs are by far the best way to run most Mongo deployments and we’ve been experimenting with various disk setups for a little over a year now (there are a shocking number of SSD options out there). Today’s release represents the future of our hosted service: each database runs in an isolation capsule with dedicated memory, IO capacity and CPU capacity. The memory/CPU/disk resources are tuned to the most common Mongo workloads we’ve seen across our 50,000 hosted databases in the last three years.

The new databases

New database are available at two price levels, the first is a “just going into production” $100/mo replica set database:

• 4GB of included SSD storage ($25/GB/mo for extra storage)
• 400MB of dedicated RAM per member

The second level offer a per/GB price break, starting at $500/mo for a replica set DB:

• 25GB of included SSD storage ($20/GB/mo for extra storage)
• 2.5GB of dedicated RAM per member

Beta autoscaling

We’re also inviting some customers to participate in a private beta test of our database autoscaling. MongoDB loves memory and IO, so we’re rolling out live resource upgrades to help customer databases stay healthy during periods of rapid growth. Adding RAM and IO (vertical scaling) is nearly instantaneous and usually a much better option than forcing your data to work well with sharding. Want in? Just add one of these new DBs to your account and let us know: support@mongohq.com

To upgrade, please send your database name to support@mongohq.com.

Run MongoDB 2.4.0-rc0 on MongoHQ today. On any MongoHQ paid plan, you can use the new 2.4.x features (full-text search, updated aggregation features, and newly optimized geospatial indexes, all backed by the V8 engine). For a refresher of MongoDB 2.4.0 features, take a look at:

• MongoDB 2.4.0 Release Notes
• Explore MongoDB 2.3 on MongoHQ
• MongoDB and Full Text Search: My First Week With MongoDB 2.4 Development Release

To upgrade, please send your database name to support@mongohq.com. We will upgrade your database as quickly as possible. The upgrade process will require a quick restart of your database; then, you will be off and running. MongoDB 2.4.0 will become the default version when the production release is made available.

To test MongoDB 2.4.0-rc0 in a non-production environment, our Sandbox plans called “Dharma Experimental” are by-default running the latest release candidate. We treat our Sandbox plans like free bread; please, have as many as you like, and keep testing.

Kristine - Mile 7

Kristine Toone with her husband, Brian, and children, Analise and Josiah.

Dusty with daughters Mary Margo and Dottie

One of our upcoming features requires low level, high performance networking (more about the actual feature later…). We originally prototyped the feature with Python, and exercised the new tool with a LOT of open TCP connections which caused some weird crashes. The crashes seemed random; it took us time to undercover the underlying snag. The fix turned out to be fairly easy, as you might expect.

The snag we encountered was due to a Linux limitation which is poorly documented and not widely known. The limitation is in the underlying Linux select() system call and as such, it applies to all programs (in whatever language) that use select.

The simplest way to listen to several sockets or pipes concurrently is with the select system call. In Python, select takes 3 arrays of file or socket objects to know what to listen to. Python documentation does not mention the library translates the arrays to bit vectors indexed by numeric file descriptors, which is implemented in C as part of the system interface. The bit vectors have fixed sizes determined by a kernel parameter, 1024 bits by default. Even if your select() call is listening on only one socket, given the socket’s .fileno() is higher than 1024, select() cannot handle the connections, and will trigger a runtime error.

The solution is to use select.epoll() instead of select.select(). Epoll does not have the 1024 file descriptor limitation and, as a bonus, is more efficient than select. When listening to a large number of sockets, Epoll is quicker because the library does not linearly scan a large bitmap result searching for sockets with available data. In Python, this does not matter since building the return array is likely to be much slower than scanning the bitmap, but high-performance server implementations should take this into account.

The Tradeoff

In Python, epoll events give OS-level numeric file descriptor numbers rather than mapping to the associated Python socket or file objects. Mapping the events manually can be tricky in situations when sockets are opened and closed in multiple application locations. OS-level file descriptors can be reused after begin closed, so the mapping must be fresh.

The classic article about high-concurrency server implementation is “The C10K problem” by Dan Kegel http://www.kegel.com/c10k.html. It is a bit out of date by now, but still worth reading for anyone working in this area.

We have seen a pattern recently with MongoDB, Ruby, and Resque (the background processor). Resque behavior combined with the Ruby driver behavior prevents background worker performance from scaling linearly. When Resque forks, the Ruby driver creates a new connection to the database. This translates into one new connection per processed job. If you are running 1000s of jobs / second, it will cause MongoDB to run less than optimal.

The quickest solution is the ‘resque-jobs-per-fork’ gem. As the name implies, Resque will run multiple jobs per fork. Instead of a 1-to-1 job to connection pattern, you will have 500-to-1 or 1000-to-1 job to connection pattern.

The Cause

As simple as it sounds, your connection pattern to your MongoDB affects your customer’s experience with your product. Good connection patterns consist of long running, persistent connections. When looking at the logs, you should only see a “connection accepted from” every 10 - 15 seconds, even on large scale deployments. Most of the time, poor connection patterns exist from the beginning of an application’s development, but they only become evident due to performance issues. These performance issues arise as databases grow in size and the application grows in usage.

With MongoDB, poor connection behavior of an application is exposed due to the following constraints: 1) per connection memory overhead and 2) read lock causing slow authentication.

Per Connect Memory Overhead

As of the 2.0 branch, each connection in MongoDB, allocates 1MB of RAM. Before 2.0, it was dependent on system ‘stack size’ settings. The following code in MongoDB shows the per connection memory usage algorithm:

1
2
3
4
5
6
7
8
9
10
11
12
13

static const size_t STACK_SIZE = 1024*1024; // if we change this we need
to update the warning

struct rlimit limits;
verify(getrlimit(RLIMIT_STACK, &limits) == 0);
if (limits.rlim_cur > STACK_SIZE) {
  pthread_attr_setstacksize(&attrs, (DEBUG_BUILD
    ? (STACK_SIZE / 2)
    : STACK_SIZE));
} else if (limits.rlim_cur < 1024*1024) {
  warning() << "Stack size set to " << (limits.rlim_cur/1024) << "KB. We
suggest 1MB" << endl;
}

mongo/util/net/message_server_port.cpp#L78

As with most everything in life, one of anything is practically nothing (think cars in traffic). 1 MB is a rounding error of modern RAM sizes. However, a modern large scale application consists of many components with many requests. These 1000s of operations per second could turn into major RAM usage if implemented incorrectly.

Given MongoDBs reliance on good RAM usage, deficient RAM usage can quickly ruin performance.

Write Lock & Slow Authentication

When using authentication, each connection and authentication action is a database query. If a database is under heavy write load, the authentication will be as slow as the rest of your queries. Thus, rapidly connecting to a database in a high-write environment authentication will have to navigate other locks for your database.

As with Memory Overhead, these deficiencies are not typically noticed until the application grows in data and usage.

Resque’s Forking Code & Ruby Driver’s Reconnect

Resque uses process forking to spawn new workers for each job resque/blob/master/lib/resque/worker.rb#L137.

1
2
3

if Kernel.respond_to?(:fork)
  Kernel.fork &block if will_fork?
else

Ruby driver reconnects mongo-ruby-driver/blob/master/lib/mongo/util/pool.rb#L240.

1
2
3
4
5
6
7

if socket.pid != Process.pid
  @sockets.delete(socket)
  if socket
    socket.close unless socket.closed?
  end
  checkout_new_socket
else

The Ruby Mongo Driver does this because managing connections between parent and child processes in Ruby is a beast. The fool-proof method is to re-initialize connections on each fork.

Resque & Ruby : The Effect

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

require 'rubygems'
require 'mongo'

@conn = Mongo::Connection.new("localhost", 27017, :pool_size => 10,
:pool_timeout => 5)
@db   = @conn['resque_connection_test']
@coll = @db['users']

puts @db.command({getLastError: 1})

1.upto(10) do
  client = fork do
    puts @db.command({getLastError: 1})
  end

  Process.wait(client)
end

The code above mimics the effects of a standard Resque worker. Each “puts” prints a different “connectionId”, thus each fork establishes a new MongoDB connection. If you are watching the MongoDB logs, you will see 11 lines containing “connection accepted from.”

Debugging In MongoHQ

Mongostat hides most performance issues due to poor connection patterns. With the example above, mongostat would show the same number of connections as your have Resque workers. By looking at the logs, you will see a new line containing “connection accepted from” for each time that Ruby forks the process.

Debugging decreased performance due to connectivity issues requires access to the Mongo logs. If you see more than a few connection attempts every few seconds, please consider a method to use longer persistent connections. You will achieve better resource usage, and better product delivery for your clients.

MongoHQ shared plans have access to real time logs from the Mongo server. Using these real time logs, you can see your connection patterns. For assistance, please E-mail support@mongohq.com.

• Resque & Ruby are not the only offenders with poor connection patterns. The stock Node.js driver tests for Replica Set status every second for each process – issuing reconnections. PHP and Apache is evil when not configured properly – the continuous building up and tearing down of workers triggers new connections.

To use, just click the “Clone / Upgrade” button in the upper right when viewing your database.

Data Size & Index Size Growth Charts

We have a dataSize growth chart associated with all databases now. As you grow, you can see where you have been and were you are going. The chart includes both dataSize and indexSize as reported by db.stats().

.explain() yourself mongo

By simply adding .explain() to the end of your mongo query, mongo will return details about how it goes about fulfilling that query. This will help you understand what indexes are being used and how many documents mongo actually sifts through to generate the result.

a contrived example

First, let’s generate a dataset… how about some people. Running the following code right in the mongo shell will create us 200k person objects to work with, each with their own birthdate, gender, and hair color. No indexes have been created (yet).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

var genderChoices = ['male', 'female'];
var hairChoices = ['black', 'brown', 'blond', 'red', 'auburn',
'chestnut', 'white'];
var birthdate = new Date();
birthdate.setYear(1970);

for (var i = 0; i < 200000; ++i) {
  birthdate.setHours(birthdate.getHours() + 1);
  db.people.insert({
    gender: genderChoices[Math.floor(Math.random() *
genderChoices.length)],
    hair: hairChoices[Math.floor(Math.random() * hairChoices.length)],
    birthdate: birthdate
  })
}

go unindexed query!

Let’s filter down to the first 10 males with a tasteful chestnut hair color and sort them by birthdate. I’ll use the pretty web interface.

Performance is already pretty bad on this query. With only 200k documents, our query takes over 100ms (which is bad enough to automatically get logged by MongoDB as a slow query).

Notice also, the cursor is a BasicCursor which means that no index was used.

The killer information that you should look for is the number of objects MongoDB scanned through fulfilling this query. I inserted 200k documents, and well, the nscanned value tells us we’re searching through every single document.

For sorted queries, also pay attention to scanAndOrder which when true means that MongoDB spent time sorting the results instead of using an index.

index race

Just for example, I’m going to add two separate indexes: one on hair and one on gender. Again, I’ll use the pretty web interface.

be careful adding indexes to production data

With an index on hair, and a separate index on gender, let’s run the query again.

I’m cutting off part of this explain() result for later, but first look at the cursor value.

The cursor is BtreeCursor hair_1 which says we’re using an index on hair. Mongo only needed to scan through 28.5k documents to complete the query. But why use the hair index?

a DB with a plan

Here is the next part of the explain output:

This shows that MongoDB tried 3 ways to fulfil my query. It tried using the hair index, the gender index, and no indexing. The poor gender and BasicCursor options only scanned through about 200 documents before MongoDB terminated them (because it already had the answer from the hair index).

So, we did better by adding the hair index. If this is the only query we ever run against our data, then the gender index isn’t helping (and actually slows down updates/inserts/deletes to maintain the index).

Looking back at the hair index, the nscanned tells us that MongoDB scanned 28.5k documents and generated a result of 14k documents that pass the query.

28.5k is about 15% of our total data, which is a nice reduction. We can do better though with a compound index which will bring the nscanned value closer to the total result set.

better, stronger, faster

Let’s create an index on hair, gender, and then birthdate. We’re choosing hair first because that’s the most efficient at breaking apart our data into smaller fragments (which is why MongoDB chose it in our index race above).

How does it perform?

The results are much faster, and in this optimal case, our nscanned is equal to n and scanAndOrder is false. Good job contrived example. You show us what we can aspire to.

Explain yourself

Explain is a great tool to understand a little more about how your query is actually working with your indexes, and with MongoHQ it’s a single pretty button away from helping you.

Customers have different types of analytics engines on our hosted MongoDB platform ranging from usage metrics, business domain specific metrics, to financial platforms. The most generic type of metrics that most clients start tracking are events (e.g. “how many people walked into my stores” or “how many people opened an iPhone application”).

A proper schema is the first step to getting off the ground quickly on a platform that will scale.

The Naive Approach

With MongoDB, the first urge is to begin inserting documents quickly. The first documents typically have the following schema:

1
2
3
4
5

{
  store_id: ObjectId(), // Object id of a store
  event: "door open", // will be one of "door opened", "sale made", or "phone calls" 
  created_at: new Date("2013-01-29T08:43:00Z")
}

To run a query on the event, store_id, and created_at, you run the following query:

1

db.events.find({store_id: ObjectId("aaa"), created_at: {$gte: new Date("2013-01-29T00:00:00Z"), $lte: new Date("2013-01-30T00:00:00Z")}})

These types of queries are deceptive. When you build the query on your local environment, it is fast. When scaling to 10 GB, they become slow. Typically, to increase speed, compound indexes are added for the following:

1
2
3

db.events.ensureIndex({store_id: 1, created_at: 1})
db.events.ensureIndex({event: 1, created_at: 1})
db.events.ensureIndex({store_id: 1, event: 1, created_at: 1} )

Each of these indexes must entirely fit in RAM. New documents do not have greater values for store_id and event than perviously inserted documents. Any insert command will add a document record to the middle of an index. Any query will have to venture into the middle of the index. Thus, all indexes must fit in RAM to maximize performance.

An Optimized Document Schema

To optimize your document schema, create a time_bucket attribute that breaks down acceptable date ranges to hour, day, month, week, quarter, and/or year.

1
2
3
4
5
6
7
8
9
10
11
12
13

{
  store_id: ObjectId(), // Object id of a store
  event: "door open",
  created_at: new Date("2013-01-29T08:43:00Z"),
  time_bucket: [
    "2013-01-29 08-hour",
    "2013-01-29-day",
    "2013-04-week",
    "2013-01-month",
    "2013-01-quarter",
    "2013-year"
  ]
}

With the optimized schema, you would create the following indexes:

1
2

db.events.ensureIndex({time_bucket: 1, store_id: 1, event: 1})
db.events.ensureIndex({time_bucket: 1, event: 1})

With this document schema, we use a practice called “bucketing”. Instead of building a query on a range, we run the query:

1

db.events.find({store_id: ObjectId("aaa"), "time_bucket": "2013-01-29-day"})

The draw back of this optimized document schema is every query must include a time_bucket attribute for every non-_id query. However, when querying most reporting systems, a date specification is required

Better Use of RAM

Using the optimized time_bucket, new documents are added to the right side of the index. Any inserted document will have a greater time_bucket value than the previous documents. By adding to the right side of the index and using time_bucket to query, MongoDB will swap to disk any rarely older documents. MongoDB runs with minimal RAM usage. Your “hot data” size will be the most recently accessed (typically 1 - 3 months with most analytics applications), and the older data will settle nicely to disk.

Neither queries nor inserts will access the middle of the index, and older index chunks can swap to disk.

Bonus Points: Using the Aggregation Framework

Using aggregation to find the number of specific events per day, we can run:

1
2
3
4
5
6

db.events.aggregate(
  {$match: {time_bucket: "2013-01-month"}},
  {$unwind: "$time_bucket"},
  {$project: {time_bucket_event: {$concat: ["$time_bucket", "/", "$event"]}}},
  {$group: {_id: "$time_bucket_event", event_count: {"$sum": 1}}}
)

Disclaimer: I used the $concat operator that will be available in MongoDB 2.4. To use a hosted MongoDB 2.4 Development Branch, take a look at our experimental databases.

When using the aggregation framework, cache a final run for any particular day to a summated collection. Use this summated collection to present data via any application or reporting server.

Follow Up Materials

The premier presentation for MongoDB Analytics is by John Nunemaker of Github. He regularly makes the rounds at MongoDB conferences covering how to do MongoDB analytics properly. 10gen makes these presentations available. MongoDB for Analytics (at Github)

When building a scaling analytics system, look for logical “buckets” for data. Avoid using “$in”, “$gte”, “$lte” operators when possible. MongoDB is fun because it rewards creativity for good schema design.

From my initial reading of 2.3 release notes, I underestimated full text searching. Before we get started on what I wanted to talk about today, I wanted to mention one gotcha.

So, the gotcha …

CopyDB and Auth - Using the hosted MongoDB 2.3 server, I wanted to use some of MongoHQ’s tools to transfer data from an existing database to my new MongoDB 2.3 database. To do this, I used the built-in copyDatabase command, which MongoHQ exposes via their web-based UI. When I did that, I saw the following:

1
2

Tue Jan 22 14:58:44.314 [conn697] copydbgetnonce is not supported when running with authentication enabled
Tue Jan 22 14:58:44.315 [conn697] copydb is not supported when running with authentication enabled

I temporarily got around this using the mongodump and mongorestore command, but it looks like there are still some auth-related bugs to be cleaned up in the developmental release. No worries as that is what developmental releases are put out there to figure out.

Now, on to some details around full-text search…

Underestimating Full Text Searching

Full-text searching with MongoDB 2.4 is more complex and powerful than originally illustrated in our first blog post outlining this feature. In the original example, we showed:

1
2

db.emails.ensureIndex({body: "text"}, {name: "email_body_text_index"})
db.emails.runCommand( "text", {search: "Pho"} )

However, there are a number of other settings that are available for these query types, including:

• Field weighting
• Negative Matches
• Phrase Matching
• Entire document indexes

So, I’ll quickly break down each one of these parameters, as each is pretty useful:

Field Weighting

The documented spec only allows one text index per collection, but it allows multiple fields on that single index. Initially, my thought goes to fields like tags, subjects, and fulltext. Using the following:

1
2
3
4
5
6
7
8

db.emails.ensureIndex({tags: "text", subject: "text", "body": "text"}, {
    name: "email_text_index",
    weight: {
      tags: 5,    // Assume folks are better at tagging than writing
      subject: 4, // Assume the subject is better than the body at description
      body: 1
    }
})

MongoDB uses a scoring algorithm to choose the most appropriate documents based on the weights and the number of matches.

Negative Terms (not yet documented) and Phrases

If you look at the code, you will see negative words are also possible. So, you run something like the following:

1

db.emails.command('text', {search: "Pho -'gangnam chicken'"})

Running the above you would find mentions of “Pho”, without mentioning ‘gangnam chicken’.

Entire Document Indexing (i.e. wildcard)

Scanning through the documentation, I ran across this nugget: “$**”. With it, you can supply the entire document to full text searching like so:

1

db.emails.ensureIndex({"$**": "text"}, {name: "email_index_text"})

If you are following along at home, you will need to drop the index above before adding this one

I am working with very large documents on our test database, and as expected the wildcard indexes were noticeably slower than specifying precise fields. This certainly makes sense, but is something to keep in mind as your data set grows.

Index Sizes

After this first week, I began thinking “how big are these indexes?” How much ram will they require? In our internal tests, we are not working with huge data sets on this project yet, but we were seeing wildcard indexes consuming about 1/3 of the total data size for the collection.

Given this, you should take this type of memory use into consideration as some long-running text searches could inadvertently shove a good chunk of your active data set out of memory. 10gen makes mention of this in their release notes, so make sure you keep this in mind as you plan out resources required to use this feature.

The Code

All of MongoDB is open-source. All of the full text search is modularized inside the fts directory at:

https://github.com/mongodb/mongo/tree/master/src/mongo/db/fts

The significant files are:

• fts_matcher.cpp contains the matching algorithm – there are some undocumented nuggets in there.
• fts_search.cpp contains the search runner and result builder.

Even if you are just a Ruby or Node or PHP or Python developer and prefer to stay away from C++, reading code is always a good exercise. As you learn more about this feature, take a moment to learn more about the code that operates it as well.

We are excited about some of the capabilities that full-text search adds to MongoDB and looking forward to seeing this feature mature over the coming months.

Get Started

To get started testing the MongoDB 2.3 release, sign up for MongoHQ, and create a database on the “Experimental” plan.

• Full-text search
• Change in the internal Javascript engine
• Hashed indexes

We’ll talk a bit more about these changes in just a moment, but first, we wanted to give our customers a way to try out the new features, even though they are in the 2.3 development release. You can do this by doing the following:

1. Log into MongoHQ
2. Create a new sandbox databases with the “Dharma 2.3 Experimental” option.
3. Insert data
4. Make profit.

Now that you are all set and have your experimental database provisioned and ready to go, let’s talk about a couple of the new features.

Full-text Search

10gen has pointed out on numerous occasions that this has been one of the most-requested features on the MongoDB roadmap, so the engineers are pretty stoked to finally offer it.

However, it is still highly experimental. Good to explore, but do not use it on production environments.

If you had a collection of emails with a field called email_body, you can add the following index:

db.emails.ensureIndex({email_body:text}, {name:"email_body_text_index")

Allow the index to build and from there, you should be good to start testing queries against it. A query may look like:

db.emails.runCommand( "text", {search: "Pho"} )

This will return a BSON document … not a cursor. So, a bit of a change there.

This should get you started … there are a number of additional features and options that you can use when querying. For more information about doing text queries with this new functionality, check out the release notes for MongoDB 2.3.

Changes to the Javascript Engine

With the 2.3/2.4 release of MongoDB, 10gen is replacing Mozilla’s Spidermonkey javascript engine with Google’s open-source V8 engine. While there are some speed improvements in various cases, the details of this change are probably a bit out of the realm of the interest of most people using MongoDB as a database.

Nevertheless, it is a worthwhile endeavor to benchmark some of your map reduce functionality against this new version of MongoDB as compared to 2.2 versions of the code.

That’s It

We hope you have fun testing this developmental release of MongoDB. Please do not use this environment for anything production-related. If you have any questions, we’re happy to help!

When making the transition from a single server setup to a Replica Set there are many more benefits than just a hot standby. Read preferences are one that should be heavily considered. Benefits vary from reducing system load and/or network throughput on the primary to location based reads that are based on latency from the client. There are several options for configuring read preferences and different reasons why you might choose one over another.

Recent versions MongoDB have included support for sending read operations to the secondary member but with the release of 2.2 three new modes were added. These include primaryPreferred, secondaryPreferred and nearest. As in prior versions these can be applied to per connection, collection or operation.

Read preference modes:

• primary

  The default. Send all read operations to primary even if primary is unavalialbe, thus failing. This would be used when stale results are not a option.

• primaryPreferred (NEW)

  Similar to the default although when the primary is unavaliable read operations are sent to the secondary. This provides some level of uptime during the transition from secondary -> primary during a failover. Although failovers don’t occur that often one must be prepared.

• secondary

  Send all read operations to secondary. If secondary is unavailble the operation will fail.

• secondaryPreferred (NEW)

  Send all read operations to secondary unless secondary is unavailble. When the secondaryu is unavailable reads are sent to the primary.

• nearest

  Read operations are sent to the nearst member using the member selection http://docs.mongodb.org/manual/applications/replication/#replica-set-read-preference-behavior-nearest process. Could be used when stale results are acceptable and low latency is a priority.

Driver examples (Ruby):

• per connection

  When specifying the read preference in the connection all future objects will inhereit this preference unless specifically set.

1
2
3

db =
Mongo::ReplSetConnection.new(['host.m0.mongohq.com:10011','host.m1.mongohq.com:10011'],
:read => :secondary_preferred).db("myapp")

• per collection

1
2
3
4
5
6
7

db =
Mongo::ReplSetConnection.new(['host.m0.mongohq.com:10011','host.m1.mongohq.com:10011'],
:read => :primary).db("myapp")
auth = db.authenticate("user","password")
staging = db.collection('staging')
staging.find({:customer_name => 'John'}, :read =>
:secondary_preferred)

Other driver documentation can be found here: http://api.mongodb.org

As you can see the new modes offer more flexibility to where your reads operation take place. Although most applications perform well without configuring read preference fine tunning for growth may solve future issues.