The general idea behind these code katas is that they are designed to be well-formed coding exercises to give programmers a chance to stretch and develop new skills. I’ve written more graph algorithms that run in a single-process space than I’d care to discuss (here’s a simple graph implementation I wrote to learn Google’s Go for instance Go-lang datastructures ), so I wanted to solve this one using a different paradigm of problem solving.

For this one I wanted to write it as a MapReduce job. I’m familiar with CouchDB and it’s built-in MapReduce implementation, but I wanted to go with something that worked in a fully-distributed mode as well as serve as practice for Hadoop. I love all my NoSQL options.

The code for this discussion, as well as instructions to setup and run, is available at : codekata19-mapreduce.

For the sake of discussion, let’s decompose the problem into two subproblems: graph construction and graph traversal.

Problem 1 – Graph construction

Given a list of words, we need to construct a graph such that every vertex is a valid word and every edge represents a valid single-letter transform to get to the next vertex. To solve this using MapReduce we make the input set of data the wordlist. The only “global” information that we need to pass around with the actual processing job is the dictionary so each JVM/Mapper in the system knows what constitutes a valid word (this is done using Hadoop’s DistributedCache mechanism). The Map function will emit a key for every valid, single-step transform we can determine as valid. So the output is essentially the edge-set of our graph.

The reduce function’s only job in this case is to make sure we don’t have duplicates and to format the data in a way such that we can use this MapReduce job’s output as the input to the next phase. The final results of this will look something like:

cat   cog,|-1|WHITE|

with an intentional trailing pipe.

The code for this MapReduce job is available here: CodeKata19.java.

Problem 2 – Graph traversal

Graph traversal in this case is a brute-force breadth-first search of the entire graph. Cailin has done an excellent writeup of parallel, distributed breadth-first search at (Breadth-first graph search using iterative map reduce algorithm). The only modification I made to the proposed setup was including a complete path after the last pipe. The algorithm will run until no more GRAY nodes exist in the network, meaning all reachable nodes from a selected start point have been reached.

It’s worth noting that instead of being the “fastest” implementation, which could easily fit inside a single-process space, this is more of an exercise in distributed algorithms. One of the key advantages is that once this completes running we have the single-source, all-destination pair shortest path result. The final result contains all shortest paths to all feasible solutions.

The code for this MapReduce job is available here: CodeKata19Search.java.

If you need help Thrift running take a look at my post on Cassandra’s thrift interface that has all the prerequisites listed.

The example assumes a table “t1″ and a column “f1″.

Rich Atkinson already has a great blog post on how to get up and running with Thrift on Snow Leopard. So if that’s what you’re running, I’m going to suggest you check it out. If you’re running Ubuntu you’ll need to satisfy the following dependencies:

sudo aptitude -q -y install libexpat1-dev libboost1.37-dev g++ autoconf automake libtool

and the source can be obtained with:

svn co http://svn.apache.org/repos/asf/incubator/thrift/trunk thrift

and then you can proceed with the standard “configure && make && make install”.

Hopefully at this point you have the Thrift native libraries installed. Since this is about Ruby, you should also install the Thrift gem that will take advantage of the native libraries.

sudo gem install thrift

Armed with both native library and gem, let’s go ahead and navigate to your Cassandra install’s interface directory (cassandra/interface) and build the ruby code:

thrift --gen rb:new_style cassandra.thrift

This will generate (as of this writing…) three files: gen-rb/cassandra.rb, gen-rb/cassandra_constants.rb, and gen-rb/cassandra_types.rb. At this point you can create a temp.rb file in the gen-rb folder to play around with connections. Here’s a short example of how to make a GET request for a specific key:

It’s worth noting that there *is* a gem available on github from fauna/cassandra that creates a much easier-to-work-with client, but since the interface for Cassandra is still evolving and changing the client is broken at the moment. As far as I know this only applies to Cassandra 0.4.1 DEV and newer. I’m very much looking forward to a working update.

It has a handful of constants at the top of the file you need to define for everything to work right. Of note are the application name and the machine’s FQDN it should answer on. If your using EC2 you might have to tweak some configuration afterward since the FQDN in DNS probably won’t match the IP of the machine’s interface.

Also, if you plan on using authorized_keys and deploying from a git repository it makes things a lot easier if you tar and gzip the relevant files and put them in an S3 bucket to pull from. The script handles this case as well.

As always, make sure you understand what a provisioning script does before you accept it with blind faith. At Flatterline we use this as our base template.

Consider the following three models which are a very basic “has and belongs to many” setup:

So as you can see, a user can be in many groups and a group can have many users, all by way of the memberships join relationship.

The two example use cases I’m going to work with are:

1. Given a user, what groups is he/she in?, and
2. Given a group, who are the members?

Both of these are pretty typical, but can yield surprisingly different results from the database’s perspective. If we try and get this data from the console we either start with a user and navigate to group, or the reverse. Here’s the MySQL EXPLAIN output from the console ( I recommend viewing the RAW output unless I can figure out how to make github display it correctly):

Totally understanding the output of the EXPLAIN syntax is well outside the scope of this post, but we’re going to need to cover the basics. The first thing you should notice is the word ALL in the type column and the NULL in possible_keys. This indicates that MySQL’s query optimizer has no index to read from and will be forced to perform a table scan to return the result. In general, this will kill your performance. Note the rows value of 100. This value will be whatever the size of your table is. If you have 500,000 records, then the database will check all 500,000 rows.

It’s worth noting that for small datasets you’ll see ALL and a possible_keys value. This means the optimizer believes that scanning the table will be faster than actually loading the index into memory. This is generally fine.

So let’s go ahead and add a composite index on [user_id,group_id]. The SQL is:

ALTER TABLE memberships ADD INDEX test_index(user_id,group_id)

Now let’s repeat the previous queries.

This is where I see most people stop when it comes to performance optimization. Note though that these tables aren’t the same! If you join from users to groups (the first query) you see a massive speedup. Only one row is consulted (instead of 100) and it’s in the index. A further benefit we see in both queries is the “Using index” in the Extra column. This means that MySQL can determine the query result without ever checking the actual table because all required info is in the index (ie no extra disk hits). Unfortunately, joining from groups to users (second query) still (sorta) sucks. It says index instead of ALL, but that just means it will have to scan the entire index rather than scan the entire table on disk. This is a marginal improvement at best, so 50% of our use cases still suck.

Here’s the explanation: B-Tree indices are unidimensional structures. That means that the interior nodes of the index tree are strongly ordered, and thus cannot be arbitrarily accessed out of order. If that doesn’t make any sense, it means that joining from users to groups is not an equivalent operation to joining groups to users because of the ordering of the index elements.

So let’s cleanup the second use case by adding an additional index, exactly like the first, except the order of the elements is reversed. Here’s the SQL:

ALTER TABLE memberships ADD INDEX test_index1(group_id,user_id)

Now we have symmetric indices. Let’s run our queries again with our second index in place:

Voila! Now it doesn’t matter which way we join the tables because we have an index that is correctly ordered based on the directionality of the join. You can even see that the optimizer selects a different index (key) depending on which direction you join the tables, exactly as expected. Also, both queries now only require consulting the exact number of rows necessary and won’t involve any further disk hits as both queries can be satisfied with data available entirely within the index.

JeoS is shorthand for “just enough operating system.” It’s a simple short install process that’s hidden in the server installation ISO. On the startup you just have to press F4 and select “Minimal virtual machine”. Rather than installing the entire Ubuntu server package you get an extremely pruned-down version that only includes the absolute bare essentials to get a server running. If your like me and believe that VMs should be created for a singular purpose only, then it will make your life even easier to create a bunch of small VMs. According to their specs you should expect something like 380MB disk images running in about 128MB of RAM.

I would like first and foremost to wish you well on your endeavor.
Starting a business is no small feat and you will certainly need all
the help you can get to bring your idea to life. That being said,
let’s talk about money.

You will require money if you plan on developing software. If you’ve
considered finding and asking a developer for his or her time in
exchange for sweat equity, I’m going to strongly advise you against
it. Not only are these requests often derided, they are often
insulting to the developer. Worse yet, they make you appear an amateur
with little understanding of how the technology field and development
works.

In the words of the Joker, “If you’re good at something, never do it
for free.” All the engineers I trust to execute a project swiftly,
professionally and with quality, especially under startup conditions,
will not touch an upfront equity deal with a 10-foot pole. They
understand not only the value of their abilities, but that they could
just as easily band together and make their own startup. A common
trait among these engineers is an understanding of their own
entrepreneurial value. Investing time in your goals, when they could
be developing their own, comes across as a high opportunity cost with
an extremely unlikely payoff. As great as you think your idea may be,
90% of startups still fail.

If you’ve ever tried to secure external business funding, whether it
be angel or venture based, then you know that there’s most likely
weeks of negotiation coupled with presentations, milestones and the
unmistakable feeling that you’re in a fish bowl being watched. This
shouldn’t come as a surprise since you’re asking someone for a large
sum of money. The lender wants to see mitigated risk as well as
receive some ownership for their money. Equity to a developer works
the same way, except it’s often approached as hiring an employee
instead of bringing on an investor. Be prepared with your business
plan, term sheet, and probably a beer. An “I’m giving you the
opportunity to get in on the ground floor” attitude is going to make
you the target of contempt.

So when is equity OK? It will vary depending on the unique situation
of the developer, but a good rule of thumb would be to not offer it
until AFTER you’ve developed a successful working relationship in
which you have demonstrated your ability to execute your business
plan. Your stock options are worth zero until you successfully exit
and, even then, only if the strike price is better than market value.
Proof of execution is what matters today, not ideas. It’s not that
somebody is working for free, it’s that they’re working for a better
long-term payoff. A payoff which you must prove is likely to happen.
Also, just like an investment, the more risk that’s on the table at
the time of the offer, the more of your company you should be prepared
to part with. If you offer a fractional percentage, don’t expect a
call back.

It’s going to be a rough road, as it is with all startups, and I wish
you the best of luck in your venture.

-Chris

Here’s a quick Ruby means to convert latitude and longitude into a timezone ActiveSupport recognizes.  This snippet only relies on Hpricot and the freely available Geonames API:

GMT offsets are a convenient way for moving time data in and out of UTC as well as for not having to deal with arbitrary string names.

If you have two nodes, nodex and nodey , and you want to be able to do something on both you can start the Erlang shell on nodex with the following:

erl -setcookie someFormOfPreSharedSecret -name nodex

and a similar command on nodey

erl -setcookie someFormOfPreSharedSecret -name nodey

The key here is to make sure that both nodes have the same magic cookie value, otherwise they won’t be able to communicate.

From nodex issue net_adm:ping(‘nodey@your_domain_here.com’).  If the connection was successful you’ll see the result pong, if something didn’t go right, you’ll see pang.  From this point, you can run:

nodes().

and see every node your currently connected with.

Now for the magic, let’s say you want to start a particular task running on all of your nodes.  We’re going to achieve this by using the lists:foldl function:

Foldl takes 3 arguments: an anonymous function to call for every element of the third parameter, an accumulator to pass to the anonymous function, and the the list of things to iterate over.  If your more familiar with Ruby this would hypothetically look like:

The key difference being Ruby doesn’t return Pids to later refer back to those concurrent processes we started.

After running the Erlang example every node will run io:format and print a simple message.  That’s really all there is to distributing Erlang processing power.  By retaining a list of the remote Pids you can do neat tricks with firing up a good number of initial processes and then sending them subsequent messages.  I’ll make a valiant effort to write about this more.

For the sake of example let’s assume we have Authors and Posts. Further, let’s assume that an Author has many Posts and that these are represented as independent documents. The goal here is to find posts created by a specific author.

In SQL, this would probably be represented as a table of authors and a table posts. Getting the aggregate result would be something along the lines of:

SELECT * from authors join posts ON authors.id = posts.author_id

This is just a simple standard JOIN, nothing very interesting about it. However, in a document oriented system this “join” operation doesn’t really exist. Enter view collation.

View collation allows us to define a map/reduce function pair that takes in multiple document types, aggregates them onto a single key value, and gives us a means to search on author id.

Let’s say we have an author document like the following:


{'type' : 'author', 'name' : 'Chris Chandler', '_id' : '22d43eaa7e06c9c37ed3e0489401a506' }


and some number of post documents similar to:


{'type' : 'post', 'title' : 'Hello world', 'Body' : 'body text', 'Author' : '22d43eaa7e06c9c37ed3e0489401a506' }


In this case our “foreign key” is 22d43eaa7e06c9c37ed3e0489401a506. The mapping function would need to connect these records based on that key is something like the following:

function(doc) {
if(doc.type == 'author')
{
emit(doc._id , doc);
}
else if(doc.type == 'post')
{
emit(doc.author, doc);
}
}


This view will generate an intermediate hash table containing entries with the author’s key. In essence we have one key (the author’s) pointing to either a post the author has created, or the author’s record itself. To make this view answer the question ‘Show me all posts created by a certain author’ we need to write a reduce function that removes the unnecessary author records so the final table will only contain author keys pointing to lists of posts.


function(keys, values, rereduce)
{
var posts = [];
for(var i = 0; i < values.length; i++)
{
if(values[i].type == 'post')
{
posts.push(values[i]);
}
}


return posts;
}


The final result set appears as:

"22d43eaa7e06c9c37ed3e0489401a506"
[{_id: "d0d0ea6de45c9f4ff983f12a9fed9008", _rev: "2624588756", body: "Weee!", title: "Hello world 2", author: "22d43eaa7e06c9c37ed3e0489401a506", type: "post"}, {_id: "9de65ae955ecc2ea35055b9339f1651c", _rev: "2347078231", body: "Weee!", title: "Hello world", author: "22d43eaa7e06c9c37ed3e0489401a506", type: "post"}, {_id: "5d1ad3eed26f84879835fd47e44f7f55", _rev: "1163133569", body: "Weee!", title: "Hello world 2", author: "22d43eaa7e06c9c37ed3e0489401a506", type: "post"}, {_id: "0717ae0da9bf5919da0957268667c3f4", _rev: "1063237208", body: "Weee!", title: "Hello world 3", author: "22d43eaa7e06c9c37ed3e0489401a506", type: "post"}]