Frank McSherry

The impact of fast networks on graph analytics, part 2

Fri, 31 Jul 2015 17:00:00 +0000

This is a joint post with Malte Schwarzkopf, cross-blogged here and at the CamSaS blog. It follows up a previous post evaluating the impact of fast networks on graph processing systems.

In the first part of this series, we have shown that some computations see large speedups when running on a faster network, illustrated by a PageRank implementation in timely dataflow in Rust. This seemed to run counter to the findings in a recent NSDI paper that many “big data” computations can only gain a 2-10% speedup from a faster network: although we found this to be true for the Spark/GraphX stack investigated in the paper, it did not generalize to timely dataflow.

We are going to try to explain a bit more about why it doesn’t generalize: how are the systems different, and what should we learn?

We also profile both systems to explain the differences that we observe, address some of our readers’ questions from part one, and show some further improvements we have since made.

Finally, we return to our initial question of whether having a 10G network is beneficial. We compare process-level aggregation and worker-level aggregation at 10G: with a fast network, reducing computation and increasing communication actually helps. However, maybe a faster aggregation strategy could bring 1G to performance parity with 10G? It turns out it cannot. We show that even if aggregation was infinitely fast, the amount of aggregated data is still sufficiently large that sending it at 1G line rate would take longer than it takes for our 10G worker-level aggregation implementation to complete an entire PageRank iteration.

tl;dr #1: GraphX spends significantly more time computing and stalled on memory accesses than it could possibly spend using the network. Timely dataflow, by comparison, let us write PageRank as a “balanced system”, bottlenecked evenly on each of its resources.

tl;dr #2: worker-level aggregation on a 10G network is 10-20% faster than the more aggressive process-level aggregation, even though it sends significantly more data.

tl;dr #3: a lower-bound analysis shows that even the best possible 1G implementation of process-level aggregation will be at least 2-3x slower than our 10G implementation.

Bonus: timely dataflow improved since we published part one, bringing the PageRank per-iteration times down to about 0.5s (from 0.7s). The PageRank code didn’t need to change to benefit. We have some of the improved numbers in paretheses, but unless otherwise mentioned the data are from the weeks-older version.

Performance comparison

To better understand the difference between timely dataflow and GraphX, we collected some utilization data from the cluster machines while running our computations. The timelines presented in the following were recorded using the collectl utility, with samples taken every 100ms on each machine.

GraphX

Lets look at a trace of GraphX on one machine over its execution on the uk_2007_05 graph. Different machines’ timelines can differ significantly with GraphX, so we also supply the graphs for all machines.

This is a bit hard to read; let’s zoom in to the GraphX plot a bit. We pick a 30 second range in the iteration phase (210-240s; graphs for all machines):

That’s a bit more readable; there are three distinct iterations clearly visible here.

The plots tell us a bit about where GraphX’s time goes:

There is some idle time during each iteration, where the Spark workers wait on a few slower (or more burdened) machines (e.g. caelum-302). It’s also evident from the gradual drop in CPU utilization that some tasks on each machine take longer than others. This corroborates results from the NSDI paper, that Java GC can be a cause of stragglers (§5.3).
GraphX uses the disk, but not very much. In the initial phase (before iteration 1 completes), the disk I/O is composed of local reads of the input data set (each of 23 read bursts reads 1.5 GB of data). Although each worker’s data are held in memory,⁰ in each iteration GraphX has two write bursts to the root filesystem SSD,¹ which (as we understand it) is a consequence of Spark writing all shuffled data to disk (see §3.2 in the NSDI paper).
GraphX’s memory usage is truly epic. The graph’s input edge list in ASCII is 63 total GB on disk² and 15 GB in binary, which when partitioned should amount to less than one GB per machine across 16 machines.³ GraphX uses substantially more memory than this: its memory footprint grows to 42 GB in the load phase, and further to 50 GB during the iterations. This is possibly due to the overheads of tracking many small Java objects (a known issue, being addressed in Spark 1.5). As a result, GraphX doesn’t run on the uk_2007_05 graph with fewer than eight of our 64 GB machines, and it doesn’t run on the twitter_rv graph with fewer than 16 machines.
There is some brief network traffic in each iteration, with two spikes that send and receive between 1 and 1.5 GB together. The spikes occur early in the iteration, once at the start of the first period of intense CPU use, and then again at the beginning of the second one. The computation appears to rarely be blocked on them (as observed in the NSDI paper).
Most of the time is spent with the allocated CPUs (8 out of 12) fully saturated, corroborating the NSDI paper’s finding that Spark/GraphX are mostly CPU-bound. It isn’t entirely clear to us what the GraphX computation is exactly doing with all this compute, although serialization overheads are proposed in the NSDI paper. Our later profiling results indicate that serialization is about 20% of the time, and that in addition to compute GraphX may be waiting on memory for a large fraction of time.

Now, let’s look at the timely dataflow implementation for comparison!

Timely dataflow

Here is the timely dataflow trace from one machine (it is fairly representative of all 16 timely dataflow traces) across its execution on the uk_2007_05 graph:

Notice the timescale: in the time it takes GraphX to do three iterations above the timely dataflow implementation has started and completed the entire computation, including loading, partitioning, and sorting the edge data and running 20 iterations.⁵

The compute is much leaner here, there is very little skew (despite no cleverness in partitioning the graph, something that many papers claim is critical!), and memory utilization is just a few gigabytes (the line is hard to see as it remains below 2 GB, under 4% of the machine’s total 64 GB RAM).

What is timely dataflow doing differently? Well, the short version is that there isn’t really all that much to do, and timely dataflow doesn’t do much more than that.

Let’s go through the moving parts and see what has to get done.

Initially: loading up data

The computation first needs to load up and exchange the edge data, so that each worker has the edges associated with the vertices it is responsible for. There is little disk I/O in this phase as we have a hot buffer cache. Each worker then “exchanges” and sorts the edges, and gets ready to start computing.

We put “exchanges” in quotes because each worker only loads the edges it will eventually require (and the whole dataset is on the machine already), so although they go through timely’s exchange channels, none actually reach the network in practice. If we were to send them, the 3.7B edges would require 29.7 GB on the wire, i.e., 1.86 GB on each of the 16 computers. This would take about 1.5s to exchange on a 10G link, or 15s on a 1G link.

As the number of processes grows, the number of edges each process must exchange and sort decreases. For 1.4 billion and 3.7 billion edges, each of 16x8 = 128 workers must process roughly 11 million (twitter_rv) and 29 million edges (uk_2007_05). Exchanging and sorting these data takes each worker just a few seconds (we use a radix sort), and it could be even faster with more effort.

Each iteration: producing, sending, and receiving updates

Once each worker receives the signal that there are no more edges due to arrive (a “notification” in Naiad terms), it is able to start producing rank updates for each of its edge destinations. This involves first preparing the src vector, by dividing each entry by its degree, scaling down by alpha, etc.; this is all just a linear scan through a vector, doing a few operations on each element.

Producing updates

The worker then walks sequentially through the list of edges, ordered by destination, and for each edge performs one random access into the src vector. These vectors are relatively compact, and get smaller proportional to the number of workers involved, hence improving locality with scale. For 61M (twitter_rv) and 105M (uk_2007_05) vertices, each worker indexes into roughly 476,000 and 820,000 elements, respectively. These aren’t huge sets, but they don’t all fit in the CPU caches. Waiting on reads from memory is one of the places that our implementation spends its time (as does GraphX, as we’ll see).

Sending updates

For each destination, the worker must also send a (u32, f32) pair, indicating a rank update for the relevant vertex. These are handed to the timely dataflow system, which maintains buffers for each destination. Once sufficiently many updates are at hand (or the worker finishes), the data are serialized: this is essentially a memcpy, as it is in many other languages that are not JVM-based (e.g., C, C++, C#), which may explain some of Spark and GraphX’s overhead. Serialized data are then put on the network by a separate thread for each socket, without blocking the worker.⁶

Receiving updates

Having sent all of the necessary updates, a worker repeatedly receives updates from other workers. Each worker removes binary data from a queue shared with the networking threads, who each demultiplex data received from the network into said queue. Each worker thread deserializes the data (again, a memcpy), and the applies the += operation indicated by the update to its src vector. Although the data could be arriving in any order, destinations are ordered small to large, and we might hope for good locality here.

Recap

A distributed PageRank implementation doesn’t need to do much more than read and accumulate values for each destination, memcpy the results a few times, and apply the updates. Timely dataflow doesn’t really do much more than help out with this, managing the partitioning of the updates (more on this in a bit), putting bits on the wire, and coordinating the workers (telling them once they’ve received all the messages for an iteration).

In a perfect world, timely dataflow computations are either running user code as written, or waiting on data from the network. If you get the “user code” part sorted out properly, communication is really the only thing left to improve.

GraphX is integrated with a larger data processing stack, and expresses graph computations by iteratively repeating higher-level dataflow operations (“join” and “group by” stages, forming a triplets view; §3.2 in the paper) and the graph as a set of parallel collections (§3.1). This approach requires a number of auxiliary data structures and makes additional passes over the data, with several optimizations aiming to reduce communication (§4.1-4.4). This, and implementation-level overheads owed to the JVM’s data representation, is a likely explanation for the memory and computation overhead it sees over timely dataflow.

Profiling

After that detour into how the implementations work on a conceptual level, let’s answer some more questions about the differences between GraphX and our implementation with the aid of some profiling.

One reasonable question to ask is where the time goes in GraphX, and what could be done to reduce this overhead. To find out, we attached the JvmTop profiler to the running GraphX version of PageRank. The following snapshot is fairly representative for what we saw during the compute phase:

JvmTop 0.8.0 alpha - 09:16:33,  amd64, 12 cpus, Linux 3.13.0-24, load avg 4.59
http://code.google.com/p/jvmtop

Profiling PID 15963: org.apache.spark.executor.CoarseGrainedE

15% (    13.14s) scala.collection.Iterator$$anon$13.hasNext()
07% (     2.55s) scala.collection.Iterator$class.foreach()
65% (     0.84s) ....esotericsoftware.kryo.serializers.FieldSerializer.ne()
97% (     0.68s) org.xerial.snappy.SnappyNative.arrayCopy()
80% (     0.64s) scala.collection.Iterator$$anon$11.next()
59% (     0.59s) io.netty.channel.nio.NioEventLoop.select()
38% (     0.55s) org.xerial.snappy.SnappyNative.rawCompress()
18% (     0.50s) com.esotericsoftware.kryo.Kryo.readClassAndObject()
82% (     0.42s) ....esotericsoftware.kryo.serializers.FieldSerializer.re()
25% (     0.29s) ....esotericsoftware.kryo.util.DefaultClassResolver.read()
95% (     0.22s) com.twitter.chill.KryoBase$$anonfun$1.apply()
81% (     0.19s) org.xerial.snappy.SnappyNative.rawUncompress()
80% (     0.18s) com.twitter.chill.ScalaCollectionsRegistrar.apply()
78% (     0.18s) ....esotericsoftware.kryo.util.DefaultClassResolver.writ()
74% (     0.17s) ....esotericsoftware.kryo.serializers.DefaultSerializers()
73% (     0.17s) com.esotericsoftware.kryo.Kryo.newSerializer()
66% (     0.15s) scala.collection.IndexedSeqOptimized$class.foreach()
57% (     0.13s) ....esotericsoftware.reflectasm.AccessClassLoader.loadCl()
46% (     0.11s) scala.collection.Iterator$$anon$11.hasNext()
40% (     0.09s) scala.collection.immutable.Range.foreach$mVc$sp()

Serialization overhead has been suspected as the main culprit, and is certainly responsible for some of the overhead. If, as a crude approximation, we sum everything with kryo and xerial in the name above, we get 19.63% of time being devoted to serialization. (Note that garbage collection, another important factor in Spark performance, does not show up here, as JvmTop only profiles when the JVM runs user code.)

However, a much larger fraction of time (72.54%) is spent in various Scala collection methods, with 57% of time in the hasNext() method of some iterator. This may be user logic (GAS vertex implementations), or edge iterators – who really knows?⁷

Additionally, one of our readers suggested taking average cycle-per-instruction (CPI) measurements for GraphX and timely dataflow in order to see which system spends more time waiting on memory. We thought that was a rather nice idea, so we went and did it using the perf profiling tool for Linux.

Performance counter profiling results.
Metric	GraphX, 10G	Timely dataflow, 10G
Cycles per instruction (CPI)	1.95	1.09
Stalled cycles per instruction	1.54	0.74
Instructions per LLC miss	252	522

The table shows what we already suspected: GraphX’s CPI is almost 2x higher than timely dataflow’s, meaning that the processor spends more time waiting on other parts of the system than actually doing work. Likewise, GraphX has a higher average number of stalled cycles per instruction, and misses in the LLC more often (about every 252 instructions vs. every 522 instructions in timely dataflow).

In essence, these numbers tell us something that the CPU timeline above did not: both timely dataflow and GraphX are memory-bound on this computation (as we expected), but timely dataflow has better cache locality and thus (potentially) spends less time waiting on memory accesses.

Good news everyone! Timely dataflow has got even faster.

Since we reported the results in part 1 of this series, timely dataflow has improved. Primarily, this has been about streamlining data movement, cutting out several redundant memory copies and simplifying the exchange channel logic, resulting in higher network utilization.

Here are the utilization timelines before and after the improvements:

The second series is effectively just a compressed version of the first: the network and compute utilizations spike higher as we need less time to process the same amount of data.

Numerically, the total and per-iteration times for 16 workers improved to 14.5s (from 19.3s) and 0.53s (from 0.75s) on the twitter_rv graph, and 19.1s (from 23.7s) and 0.59s (from 0.76s) on the uk_2007_05 graph. The per-iteration times improved by 1.37x and 1.28x for the two graphs (~30%), which bump our best-case improvement over GraphX from 17x to 23x.

We are pointing this out because we had to get some new measurements we hadn’t previous taken, and needed to re-calibrate for them. Specifically, …

Process-level vs. worker-level aggregation on 10G

As one astute reader observed, while we pointed out that worker-level aggregation outperforms process-level aggregation on a 10G network, we did not report the detailed numbers in part one.

For bonus credit, we got utilization timelines for worker-level and process-level aggregation on 10G to visually present the trade-offs involved. These measurements are on the improved timely dataflow code.

The process-level aggregation experiment shown in the time series shown here uses the network noticeably less than the worker-level time series does, but also takes longer because it keeps the workers busy for a bit longer on each iteration (due to the extra aggregation work). As you may remember from the first part, the worker-level aggregation strategy allows us to start sending updates as we produce them, while process-level aggregation must wait until all updates are produced.

When we applied our trusted perf profiler, we found that both aggregation strategies consume about the same number of cycles, but the worker-level aggregation spends these cycles managing the additional network data on separate cores, rather than occupying the worker threads with additional aggregation. Moreover, we found that worker-level aggregation hits the last-level cache twice as often (probably due to the network threads accessing a larger working set), but it has a lower LLC miss ratio (39.6% vs. 47.1% for process-level aggregation). Finally, worker-level aggregation has a higher CPI (1.18) than process-level aggregation (0.99) – again, likely because of its larger working set.

All in all, worker-level aggregation is 10-20% faster than process-level aggregation on the 10G network, but up to 4x slower on 1G (as discussed in part one). This comes despite sending more data over the network – sometimes, being more chatty actually helps!

Lower bound analysis

That process-level aggregation stuff sure seems to transmit a lot less data. Maybe if only we had worked harder to optimize the CPU use we would see a 1G network overtake the worker-level 10G implementation. Could a better implementation get closer to the 10G performance with more engineering? Yes, but not too much closer.

Let’s reason about how fast the best possible 1G implementation could go. Due to the way we have arranged the computation, each machine must communicate at least some number of updates. Using worker-level aggregation, each process must transmit a number of updates equal to the sum of the numbers of distinct destinations for each worker. Using process-level aggregation, by contrast, each process must transmit a number of updates equal to the number of distinct destinations across all of its workers.

This does reduce the number of required updates, but it only reduces it so much.

Each process must transmit at least this smaller number of updates. The time it takes to put this data onto a 1G network is the lower limit on how fast it can go, even if the aggregation takes no time at all. If this lower limit ends up taking longer to send the data than a 10G implementation (spoiler: it will), we have learned something.

Let’s look at the minimum number of updates that one specific process⁸ must transmit when using process-level aggregation. Each update is at least 8 bytes (a u32 and a f32); divided by 1 Gb/s, we get a lower bound on how quickly this approach can iterate on a 1G network (assuming infinitely fast aggregation). Depending on the number of machines, this time is between 1 and 5 seconds (third column). This best-case bound is still often slower than our measured, real-world 10G times. If we compare the two, we get a lower bound on the speed-up from using our 10G implementation as compared to the most efficient 1G implementation possible (fourth column).⁹

Minimum numbers of 8-byte updates process zero must send each iteration for the twitter_rv graph.
cores	updates	1G lower bound	1G bound / 10G measured
2x8	31,215,505	2.00s	0.85x
4x8	26,835,975	1.72s	1.15x
8x8	22,809,014	1.46s	1.39x
16x8	15,836,150	1.01s	1.35x

Minimum numbers of 8-byte updates process zero must send each iteration for the uk-2007-05 graph.
cores	updates	1G lower bound	1G bound / 10G measured
2x8	75,220,268	4.81s	2.08x
4x8	55,465,829	3.55s	2.30x
8x8	39,443,730	2.52s	2.36x
16x8	26,866,965	1.72s	2.26x

These numbers indicate that one cannot hope to get the observed 10G performance on a 1G network just by optimizing the computation. Of course, there are other options – such as laying out the graph data along the “smarter” Hilbert-curve mentioned earlier – which we leave for another post.

How “close to the metal” are we?

With graph processing systems, hardly a systems conference seems to go by without another paper announcing mani-fold speedups over some precursor system. How much more room is there for improvement in our code?

First, we could change the program to look more like the smarter single-threaded code, which uses a Hilbert space-filling curve to lay out the edge data. This can improve cache locality and reduce time blocked on memory reads, and can also reduce the data transmitted across the network. That would be a great thing to try out, and we will probably do that, but let’s look at some anecdotal evidence of how “close to the metal” what we already are.

Perhaps surprisingly, one of the more expensive parts of our implementation is the division by workers, which each worker needs to do for each received (y, update) pair:

// repeatedly receive data from other workers
while let Some((iter, data)) = input2.pull() {
    notificator.notify_at(&iter);
    for x in data.drain_temp() {
        src[x.0 as usize / workers] += x.1;
    }
}

If we hard-wire workers to a compile-time constant power of two (specifically, 16x8 = 128), the per-iteration runtime drops by about 10%, as the compiler turns the division into a shift. The fact that replacing arbitrary divisions with constant shifts yields such speedups likely suggests that we’re getting pretty “close to the metal”, and that relatively few other system overheads exist in our implementation.

Caveats & discussion

This blog post would not be complete without some discussion of the caveats of what we’ve discussed – be they perceived or real. Here are some objections to our analysis that you might have.

Your approach seems to require a lot of hand-coding and tuning; is it user-friendly?

If you look at our code one thing becomes clear: the implementation indeed looks more like a “program” than implementations for other data processing systems might. Timely dataflow mandates relatively little program structure (compared to, say, MapReduce’s or Spark’s paradigms), which gives us the flexibility to write the program as we see fit.

This flexibility is helpful, but it comes with greater responsibility. While a programmer can more easily write performant code in timely dataflow, she can also more easily stumble into anti-patterns. GraphX’s higher-level interfaces may be the better option for writing the simplest possible programs; however, timely dataflow still maintains the core benefit of MapReduce, Spark and other parallel programming frameworks: it alleviates the user of having to implement tricky distributed coordination code, and the computational core of a PageRank implementation is pretty concise.

One could certainly write a “library” of PageRank and other graph computations atop timely dataflow, providing an appealingly simple interface to the more casual user. Timely dataflow allows the inter-operation of pre-fabricated parts (similar to Naiad vertices) with hand-rolled operators as appropriate for the computation.

Doesn’t the lack of fault tolerance mean that you avoid overhead?

One major difference between timely dataflow and Spark/GraphX is that the latter can tolerate worker failures.¹⁰ Fault-tolerance absolutely introduces overheads, both directly and indirectly: GraphX writes data to disk, but also needs to be structured as a stateless dataflow which prevents (or at least complicates) the maintenance of indexed in-memory state, while timely dataflow is stateful.

A different question is whether fault-tolerance is serving its purpose here. In batch processing, fault-tolerance is fundamentally a performance optimization, meant to get you the result faster than re-running the computation from scratch. Arguably, this investment makes sense when dealing with jobs over many hours and hundreds of machines, but if the entire computation takes on the order of 20s over a handful of machines, it may not be worth it. Of course, this trade-off may chance for larger graphs and more PageRank iterations.

That said, timely dataflow could be extended with support for fault tolerance. A blunt approach would simply check-point intermediate results (e.g., every 100 iterations). There is also some existing work on cleverer fault-tolerant abstractions of Naiad’s version of the timely dataflow model, and this could be applied to our implementation.

GraphX is part of a general-purpose stack, not a special-purpose graph processing system; isn’t this is a flawed comparison?

We weren’t specifically trying to do a bake-off between Spark/GraphX and our code, though the observed results are a happy coincidence. Rather, we were trying to understand what is it about systems that prevent them from benefiting from modern networking kit.

Spark and timely dataflow are both general-purpose data-parallel systems, and you can build graph processing infrastructure on either (GraphX and Naiad’s GraphLINQ, respectively). However, their relative performance reveals a bit about the properties of the underlying systems. In this work, we wrote code that fills GraphLINQ’s role atop timely dataflow in Rust (although we didn’t implement everything GraphLINQ supports).

What about other state-of-the-art systems; are you as fast as GraphLab or Naiad?

Performance-wise, the closest competitor (and to our knowledge, the previous-best system) is Naiad, which is based on the same timely dataflow paradigm as the code presented here. Naiad runs one iteration of PageRank on the twitter_rv graph in ≈2s using 16 dedicated eight-core machines, and in ≈1s using 49 machines (Fig. 7(a), §6.1 in the paper).

The Naiad numbers are on a 1G network, and were totally communication bound: there was about 0.15s of computation each iteration, and the rest waiting on communication. Naiad would likely see substantial improvement from a 10G network, too. We have established above that our implementation shown here cannot beat 1s on a 1G network without fundamentally changing the algorithm (Naiad used a Hilbert space-filling curve to partition the edges); on a 10G network, our best result for the same computation is 0.5s per iteration using 16 machines.

The GraphX paper authors found its performance equal to GraphLab and Giraph; our results also beat the published numbers from these systems, though we haven’t done a deep-dive comparison against them. Our understanding is that the GraphLab numbers have improved substantially from their published paper.

The NSDI paper isn’t about PageRank; how does your analysis apply to their conclusions?

The NSDI paper made some specific observations, and suggested that they may generalize. Our point is that they do not generalize too much, and that the outcome depends on the workload, the data processing system implementation and the available hardware.

The majority of the workloads in the NSDI paper are TPC-DS and BDBench queries, whose Spark implementations shuffle less data than they read as input. This property is one of the reasons invoked to explain why improving the exchange performance does not improve overall performance.¹¹

The NSDI paper’s conclusions may generalize for other types of computations, but we won’t know without implementing them as well.¹² At least one other paper reports that for well-tuned implementations, the JOIN processing common in analytics workloads should be network bound, even on an Infiniband network.

Lessons learned

To finish up, let’s recap what we’ve learned.

Computers are surprisingly fast!

Modern hardware can do quite a lot of work in a short time – for example, it can iterate over 3 billion edges and exchange hundreds of millions of updates in less than a second. However, to get this performance, it is necessarily to carefully think about how every layer of the system works. Our ability to speed up PageRank by up to 16x (or 23x, with the new numbers) suggests that there’s still a fair bit of this kind of optimization to be had in distributed data processing systems.
Faster networks can help, and they influence implementations.

Strategies that are helpful on a (slow) 1G network, such as process-level aggregation, can actually be harmful on a faster 10G network. Spark and GraphX are examples of systems designed with the assumption that I/O is slow, and that doing extra computing to reduce the I/O required is always beneficial. As we’ve shown with the aggregation strategies, this is not necessarily the case. This motivates thinking about whether we need systems capable of dynamically adapting their strategies to the hardware they’re running on.
Distributed processing does help even communication-heavy graph workloads.

Our best distributed PageRank is 11x faster than even the smart, Hilbert-curve-based single-threaded implementation (now 13.4x), and 5x faster than our best multi-threaded, single machine implementation. In other words, even communication-heavy workloads on data sets that fit into a single machine can get significant benefit from a distributed implementation, provided it makes good use of available system resources.
Balanced system design is key.

Our implementation overlaps communication and computation, stressing CPU and network resources roughly equally. In general, when adding to one resource failed to deliver speedups, we could realize them by improving on another: when we found that network utilization was uneven, we improved it by re-structuring the computation to overlap communication more, and when computation became a bottleneck, we reduced it by communicating more data. The goal should be a system in which improving any single resource won’t yield a speedup on its own. Spark/GraphX, by comparison, looks highly CPU-bound, and needs some heavy optimization before becoming network-bound.
Timely dataflow is quite a neat paradigm.

A lot of our freedom to optimize things is owed to the flexibility of timely dataflow, which only mandates some very light progress tracking, and that workers respect the promise to send no more data for a timestamp after it has been “notified”. Since this paradigm does not mandate many implementation choices, we are able to implement things such that all system resources are best utilized at all times.

While our timely dataflow implementation surely isn’t the last word in graph processing, it really is pretty simple, it gets the job done, and it goes pretty fast. You could certainly improve many of the implementation choices we made, or do something totally new, different and better! It really isn’t that hard to write this stuff, and we’d love to see more people doing inventive work rather than doing minor tweaks to existing monolithic stacks.

Maybe let’s all agree that we can stop talking so much about numbers that are slower than those above, and focus our attention on how to build things that are better.

⁰ – GraphX’s default “storage level” for the edge and vertex RDDs is MEMORY_ONLY (see here), and we use the default settings.

¹ – This is because Spark by default uses /tmp for its local intermediate storage (the SPARK_LOCAL_DIRS environment variable); as recommended, this is located on the SSD in our setup. The input data set, however, is on a spinning disk, as the SSD is not large enough to hold it (hence the “low” utilization in the load phase).

² – The ASCII edge list representation is actually a tremendously wasteful representation of the graph. For timely dataflow, we use a binary adjacency list representation that stores the same graph in 15 GB (4.2x more efficient). GraphX could, of course, use the same binary representation (but the GraphLoader class used by the canonical PageRank example assumes an ASCII edge list). This difference – and any one-off transformation into binary – only affects the one-off startup cost, however, and has no impact on the per-iteration runtime (as by then, the graph is stored as binary integers in memory).

³ – Our binary representation uses 32-bit unsigned values, which are sufficient for the graphs we study (just). Larger graphs require 64-bit values, but should still remain within 4 GB of working set per machine. Moreover, using 64-bit values potentially increases the amount of network traffic required, which would exacerbate our findings.⁴

⁴ – Frank doesn’t like footnotes. Malte did this to you.

⁵ – You might wonder why our load phase is much shorter, and why there’s little disk activity: this is because we ran all of our experiments on a hot buffer cache (i.e., having run the computation before to warm it up). GraphX does not seem to benefit much from this, which may be because its large runtime memory consumption ends up evicting most of cached data before the computation completes. However, even paging the data from a cold buffer cache ends up taking only tens of seconds, since each worker only reads less than 1 GB.

⁶ – The network threads are the reason why we allocated only eight cores per machine to the compute workers; however, as the timeline shows, we actually still have some CPU headroom.

⁷ – Footnote 7 was removed, but renumbering in HTML is too painful.⁷

⁸ – We show numbers from process 0 here, but the numbers for others are similar.

⁹ – We use our older, slower numbers to avoid re-running everything with the new code; the ratios would only improve with our newer numbers.

¹⁰ – Although when Spark workers failed due to running out of memory in our experiments, the PageRank computation did not seem to recover: it spun on the same execution stage for 30 minutes before we terminated the job.

¹¹ – Incidentally, this property is true of the GraphX PageRank too for up to three iterations: for the uk_2007_05 graph, it reads 63 GB of ASCII text, and then shuffles 23.8 GB of data per iteration. This does, however, illustrate a potential pitfall with the use of “input bytes” as a metric: the same input data can be represent at very different levels of efficiency, and may yet still result in the same runtime shuffle size. As we discussed in footnote 3, the input size can be reduced significantly by using a binary representation, without affecting the shuffle size.

¹² – We implemented TPC-H query 17 in timely dataflow to show that it can be done, and it went quite fast when we tested it. However, we haven’t done a detailed evaluation of it.

The impact of fast networks on graph analytics, part 1

Wed, 08 Jul 2015 17:00:00 +0000

This is a joint post with Malte Schwarzkopf, cross-blogged here and at the CamSaS blog.

tl;dr: A recent NSDI paper argued that data analytics stacks don’t get much faster at tasks like PageRank when given better networking, but this is likely just a property of the stack they evaluated (Spark and GraphX) rather than generally true. A different framework (timely dataflow) goes 6x faster than GraphX on a 1G network, which improves by 3x to 15-17x faster than GraphX on a 10G network.

I spent the past few weeks visiting the CamSaS folks at the University of Cambridge Computer Lab. Together, we did some interesting work, which we – Malte Schwarzkopf and I – are now going to tell you about.

Recently, a paper entitled “Making Sense of Performance in Data Analytics Frameworks” appeared at NSDI 2015. This paper contains some surprising results: in particular, it argues that data analytics stacks are limited more by CPU than they are by network or disk IO. Specifically,

“Network optimizations can only reduce job completion time by a median of at most 2%. The network is not a bottleneck because much less data is sent over the network than is transferred to and from disk. As a result, network I/O is mostly irrelevant to overall performance, even on 1Gbps networks.” (§1)

The measurements were done using Spark, but the authors argue that they generalize to other systems. We thought that this was surprising, as it doesn’t match our experience with other data processing systems. In this blog post, we will look into whether these observations do indeed generalize.

One of the three workloads in the paper is the BDBench query set from Berkeley, which includes a “page-rank-like computation”. Moreover, PageRank also appears as an extra example in the NSDI slide deck (slide 38-39), used there to illustrate that at most a 10% improvement in job completion time can be had even for a network-intensive workload.

This was especially surprising to us because of the recent discussion around whether graph computations require distributed data processing systems at all. Several distributed systems get beat by a simple, single-threaded implementation on a laptop for various graph computations. The common interpretation is that graph computations are communication-limited; the network gets in the way, and you are better off with one machine if the computation fits.⁰

These two positions – (i) “PageRank cannot be improved by more than 10% via a faster network”, and (ii) “graph computations such as PageRank are communication-bound, and benefit from very fast local communication” – didn’t jive with each other, and raised a bunch of questions: are these computations CPU-bound, or communication-bound, or is it perhaps more complicated than that? And shouldn’t it still be possible to make multiple computers connected by a decent network go faster than one computer?

We are going to look in to the performance of distributed PageRank using both GraphX (a graph processing framework on top of Spark) and timely dataflow, on a cluster with both 1G and 10G network interfaces.

tl;dr #1: Network speed may not matter with a Spark-based stack, but it does matter to higher-performance analytics stacks, and for graph processing especially. By moving from a 1G to a 10G network, we see a 2-3x improvement in performance for timely dataflow.

tl;dr #2: A well-balanced distributed system offers performance improvements even for graph processing problems that fit into a single machine; running things locally isn’t always the best strategy.

tl;dr #3: PageRank performance on GraphX is primarily system-bound. We see a 4x-16x performance increase when using timely dataflow on the same hardware, which suggests that GraphX (and other graph processing systems) leave an alarming amount of performance on the table.

Obviously, take all these conclusions with a grain of salt: we will discuss some caveats and design decisions that some may disagree with. However, our code is available alongside instructions, so you may try it for yourself!

Overview

We set out to understand the bottlenecks in a non-trivial computation that we understand pretty well: PageRank. To be clear, PageRank is not some brilliant computation, but it is more interesting than a distributed grep. It is a good example of computations that exchange data, aggregate data, and benefit from maintaining indexed data resident in memory. It can also be implemented in different ways, and thus helps tease out how well (or badly) a particular approach fits with a particular system.

This was also a good opportunity to try out timely dataflow in Rust, which is both a port to Rust and an extension of the timely dataflow paradigm in Naiad. Timely dataflow in Rust had only been run on a laptop so far, so this was a good chance to shake some bugs out.

Additionally, the Cambridge Computer Lab has a new “model data centre”, which is a modern cluster equipped with 10G networking kit, and we wanted to see how fast this Rust code can go. As it turns out, it now moves pretty briskly. You’ll see.

PageRank

PageRank is a not-wildly-complicated graph computation: the idea is that each vertex starts with some amount of real-valued “rank”, which it repeatedly shares along the directed edges to its neighbors. If one keeps doing this for long enough, the real-valued ranks start to stabilize.

Here is a straightforward, serial PageRank implementation in Rust:

fn pagerank<G>(graph: &G: Graph, vertices: usize, alpha: f32)
{
    // mutable per-vertex state
    let mut src = vec![0f32; vertices];
    let mut dst = vec![0f32; vertices];
    let mut deg = vec![0f32; vertices];

    // determine vertex degrees
    for (x,_) in graph.edges() { deg[x] += 1f32; }

    // perform 20 iterations
    for _iteration in (0 .. 20) {

        // prepare src ranks
        for vertex in (0 .. vertices) {
            src[vertex] = alpha * dst[vertex] / deg[vertex];
            dst[vertex] = 1f32 - alpha;
        }

        // do the expensive part
        for (x,y) in graph.edges() { dst[y] += src[x]; }
    }
}

If we look at the code, the computation manipulates per-vertex state (preparing src[vertex] and dst[vertex], using deg[vertex]), and then swings through graph, increasing dst[y] by src[x] for each graph edge (x,y).

The only part of this computation that makes parallelization difficult is updating dst[y]: because the edges may link any pairs of vertices, we don’t have an a priori partitioning of responsibility for these updates. So we’ll have to do that.

A Distributed PageRank

We want to map this computation across multiple workers (threads, processes, or computers), and fortunately there are several fairly simple ways to do this. The most common approach is to partition responsibility for each vertex across the workers, so that each worker is responsible for a roughly equal number of vertices.¹ We will assign the responsibility for processing vertex v to worker v % workers.

To partition the computation among workers, we also need to partition the relevant state (inputs and intermediate data) as well. The input to the computation is just a set of edges, so we must make sure that edge (x, y) makes its way to worker x % workers.

Once we have distributed all the edges to the workers, the PageRank computation is just a matter of repeatedly applying the per-vertex updates, which each worker can do independently, and then determining and communicating the updates to the ranks of other vertices.

Each worker prepares a message of the form (y, src[x]), indicating the intended update, rather than directly applying += to the ranks. These updates are then exchanged between the workers, i.e., updates to a vertex y are sent to worker y % workers.

Implementation #1: Send everything

This will be our first implementation, which we will quickly discard as being hilariously inefficient. For every edge (x, y) at a vertex, we prepare a message (y, rank) indicating interest in performing a += operation for the edge destination y. We exchange all of these messages, sending each to the worker in charge of the recipient vertex.

The figure below illustrates how this would proceed in a setting with four workers (w₀ to w₃) in two processes (P₀ and P₁).

However, this approach needs to send some data across the network for every edge in the graph. That would certainly result in some serious communication,² and 10G would look pretty good compared to 1G. However, we would only be able to conclude that a fairly naïve implementation is communication-bound. Let’s see if we can do better.

Implementation #2: Worker-level aggregation

Recall that each worker manages multiple vertices. Their edges may well have destinations in common, so each worker can accumulate the messages for each destination, and send just one message for each distinct destination. The worker can accumulate these updates into a hash table, or a large sparse vector (proportional to the number of vertices in the entire graph), but there is a much simpler way.

Each worker groups its edges by destination, rather than by source. By so doing, a worker can iterate through its destinations, accumulate its updates from each source rank, and then issue one update for the destination.

The single-threaded interpretation of this code might look like this:

// replaces "for (x, y) in graph.edges() { dst[y] += src[x]; }"
for (y, xs) in graph_transpose.edges() {
    dst[y] += xs.map(|x| src[x]).sum();
}

Because each worker only has edges from vertices x equal modulo workers, each value of x / workers is distinct. We can compact src by a factor of workers, accessing src[x / workers] instead of src[x]. Compared to the source-grouped approach, the code does random access into a small vector src rather than a large vector dst, which benefits locality.

The figure below illustrates this: each worker merges updates for the same destination (bold number) into a single update. As a result, this example four-worker setup ends up exchanging only 12 messages, instead of 16 in the naïve version.

Moreover, this approach also produces outputs as it goes, which allows workers to overlap communication with communication: a worker can start telling other workers about its updates right away, using the network at the same time as the rest of the computer.

Implementation #3: Process-level aggregation

The worker-level aggregation implementation (#2, above) is our best-performing implementation for a 10G network, but it still sends quite a bit of data.³ To more fairly compare 1G and 10G networks, we can aggregate a bit more aggressively and move aggregation to the process level to further reduce the amount of data transmitted. This, however, comes at the expense of more computation and more synchronization, since process-level aggregates across multiple workers must await all data from each worker. This reduces the overlapping of computation and communication that we can do, but we send less data overall.

Again, the figure illustrates this: after we have aggregated updates at the worker level, we also aggregate them at the process level. In the example, this reduces the number of messages from 12 to 9.

We could potentially try and make this implementation even smarter, cutting computation and overlapping communication. However, as we will show in part two of our investigation, we cannot expect a 1G network to win out over 10G even with infinitely fast aggregation.

Evaluation

Let’s see how well our implementations work!

We evaluate the time to do twenty iterations of PageRank on the CamSaS cluster,⁴ using Spark/GraphX 1.4 and our timely dataflow implementation.

We use two graphs, those used by the GraphX paper: a 1.4B edge graph of Twitter followers (twitter_rv), and a 3.7B edge graph of web links (uk_2007_05). The uk_2007_05 graph is also the one used for the results shown in the NSDI slide deck.

Some baseline measurements

Before we start, let’s think about existing baselines that should give some context.

Below, we show previously reported measurements from Spark and GraphX, as well as the runtime of GraphX on our cluster, and the runtime of two single-threaded implementations (from the COST paper).

Twenty pagerank iterations, baseline measurements.
System	source	cores	twitter_rv	uk_2007_05
Spark	GraphX paper	16x8	857s	1759s
GraphX	GraphX paper	16x8	419s	462s
GraphX	measured on our cluster	16x8	334s	362s
Single thread (simpler)	COST paper	1	300s	651s
Single thread (smarter)	COST paper	1	110s	256s

So far, nothing new: the measurement on our cluster confirms that the numbers from the GraphX paper can be reproduced,⁵. Moreover, the laptop performs pretty well even though it’s using only one CPU core, rather than 128. On the twitter_rv graph, the single-threaded implementation always beats the distributed implementations, and on the uk_2007_05 graph, the simple single-threaded implementation is only ~50% slower than GraphX (at 128x less resources!). The smarter single-threaded implementation with a Hilbert space-filling curve graph layout always beats the distributed systems by between 30% and 3x.

Is that bad news for distributed graph processing generally? Let’s see.

A timely dataflow implementation

Let’s take our data-parallel implementation out for a spin. We’ll start with just a single machine, and move from one core to multiple cores. We measure the total elapsed time (first graph), and the average per-iteration time of the last ten iterations (second graph). For reference, we also show the results for GraphX, and the simple single-threaded implementations (as horizontal bars).

Twenty pagerank iterations on one machine, multiple threads.
System	cores	twitter_rv	uk_2007_05
Timely dataflow	1	350.7s (11.33s)	442.2s (8.90s)
Timely dataflow	2	196.5s (6.39s)	297.3s (5.67s)
Timely dataflow	4	182.4s (6.12s)	192.0s (3.78s)
Timely dataflow	8	107.6s (3.70s)	137.1s (3.29s)
Timely dataflow	12	95.0s (3.32s)	114.5s (2.65s)

Well, this is good: with one thread, we still perform similarly as GraphX at 128, and we outperform the simple single-threaded measurement with just two threads⁶, and we outperform the smart single-threaded measurement with eight threads.⁷

This suggests that parallelism inside a single machine, at least, helps make this computation go faster. Maybe multiple computers, with all that networking in between, end up being slower though?

Multiple computers: 1G vs. 10G

Let’s now see what happens when we distribute the computation over multiple computers. Here we have the choice of using either a 1G network interface or a 10G network interface; we will measure both, revealing the performance gains that 10G brings (if any).

In addition to the worker-level aggregation implementation from above, we also include measurements for process-level aggregation on a 1G network (labelled “1G+”). This is more representative of a 1G-optimized implementation, and matches what GraphX does.

For each configuration, we again report the elapsed time to perform twenty iterations (first graph), and the average of the final ten iterations (second graph). Because GraphX and our implementation have different one-off start-up costs, the average iteration time when the computation is running is probably the fairest metric for comparison.

Elapsed and (per-iteration) times for twenty PageRank iterations on multiple machines using the `twitter_rv graph`, comparing 1G and 10G networks
System	cores	1G	1G+	10G	10G speedup over 1G+
					total	per-iteration
Timely dataflow	1x8	107.6s (3.70s)	107.6s (3.70s)	107.6s (3.70s)	–	–
Timely dataflow	2x8	115.2s (4.66s)	89.0s (3.51s)	65.6s (2.34s)	1.36x	1.50x
Timely dataflow	4x8	149.4s (6.77s)	80.9s (3.33s)	40.6s (1.49s)	1.99x	2.23x
Timely dataflow	8x8	145.4s (6.60s)	66.5s (2.86s)	27.6s (1.05s)	2.41x	2.72x
Timely dataflow	16x8	169.3s (7.51s)	51.8s (2.30s)	19.3s (0.75s)	2.68x	3.07x
GraphX	16x8		354.8s (13.4s)	333.7s (12.2s)	1.06x	1.10x

Elapsed and (per-iteration) times for twenty PageRank iterations on multiple machines using the `uk_2007_05` graph, comparing 1G and 10G networks
System	cores	1G	1G+	10G	10G speedup over 1G+
					total	per-iteration
Timely dataflow	1x8	137.1s (3.29s)	137.1s (3.29s)	137.1s (3.29s)	–	–
Timely dataflow	2x8	173.3s (6.82s)	135.8s (4.82s)	80.7s (2.31s)	1.68x	2.09x
Timely dataflow	4x8	231.9s (9.06s)	119.1s (4.67s)	51.4s (1.54s)	2.32x	3.03x
Timely dataflow	8x8	196.4s (8.87s)	80.1s (3.18s)	34.1s (1.07s)	2.35x	2.97x
Timely dataflow	16x8	231.2 (10.25s)	53.9s (2.13s)	23.7s (0.76s)	2.27x	2.80x
GraphX	8x8		666.8s (14.40s)	682.6s (15.00s)	0.98x	0.96x
GraphX	16x8		361.8s (9.30s)	357.9s (8.30s)	1.01x	1.12x

Phew, that’s a lot of data! There are a few important observations that we can draw from them, though:

Making the network faster does not improve GraphX’s performance much (at most 10-12%), which confirms the observations of the NSDI paper.
Making the network faster does improve timely dataflow’s performance (by 2-3x), which limits the generality of the conclusions of the NSDI paper.
Communication cost does not dominate GraphX’s performance, but it does dominate the timely dataflow performance, meaning graph computations can be communication-bound on a good implementation.
Nevertheless, performance strictly improves as we add machines, invalidating the claim that distributed systems will not be faster than a single machine when running a communication-bound graph computation on a graph that fits into the machine.
Worker-level aggregation has the opposite impact on a 10G network and a 1G network: on 10G, it yields improvements, but on 1G, it makes things slower. Moreover, at 10G, worker-level aggregation outperforms process-level aggregation (not shown in graphs), despite sending more data.
GraphX scales well from eight to sixteen machines on the uk_2007_05 graph, but as the COST paper has suggests, this scalability could just come from parallelizable overheads.
Timely dataflow on a 10G network outperforms GraphX by up to 16x on the per-iteration runtime using 16 machines (16.27x for twitter_rv, and 10.92x for uk_2007_05).

Those results allow us to answer our questions from the beginning, as we will see now.

Conclusion (for the moment)

This concludes the first part of our investigation. Apparently both 10G networks and distributed data processing systems are useful for PageRank. Phew. Existential crisis averted!

As we have seen, the three implementations (GraphX and the two timely dataflow ones) have different bottleneck resources. GraphX does more compute and is CPU-bound even on the 1G network, whereas the leaner timely dataflow implementations become CPU-bound only on the 10G network. Drawing conclusions about the scalability or limitations of either system based on the performance of the other is likely misguided.

Fast 10G networks do help reduce reduce the runtime of parallel computations by significantly more than 2-10%: we’ve seen speedups up to 3x going from 1G to 10G. However, the structure of the computation and the implementation of the data processing system must be suited to fast networks, and different strategies are appropriate for 1G and 10G networks. For the latter, being less clever and communicating more sometimes actually helps.

Distributed data processing makes sense even for graph computations where the graph fits into one machine. When computation and communication are overlapped sufficiently, using multiple machines yields speedups up to 5x (e.g., on twitter_rv, 1x8 vs. 16x8). Running everything locally isn’t necessarily faster.

Moreover, we’ve shown that we can make PageRank run 16x faster per iteration using distributed timely dataflow than using GraphX (from 12.2s to 0.75s per iteration). This tells us something about how much scope for improvement there is even over numbers currently considered state-of-the-art in research!

In part two, we will follow up with a more in-depth analysis of why timely dataflow is so much faster than GraphX. We will look at their respective resource usage over the runtime of the computation, and investigate via a lower-bound analysis whether it is at all possible to make a 1G network outperform a 10G one by doing cunning aggregation (spoiler: it’s not!).

Come back for more in a few days!

⁰ – This view is even common with those who build distributed graph processing systems: a few months ago, Reynold Xin (GraphX co-author, now at Databricks) indicated that, for communication-bound graph workloads that fit into a machine, there is no point in going distributed. That’s a somewhat surprising position: surely, we would expect a distributed graph processing system (such as GraphX) and a decent network to outperform a laptop on graph problems? Maybe Reynold is saying that “systems like GraphX are designed for graphs that are too large to fit into a single machine; there is no benefit to using them on ones that fit into a machine”. That’s a reasonable view to take, although the fact that all known GraphX results are on such graphs suggests that systems need to be evaluated using larger graphs. In either case, we later show that a distributed system does outperform a single machine even for graphs that do fit into the machine.

¹ – Some graph processing systems (e.g., PowerGraph) use far more complex and cunning schemes in order to balance the amount of work each worker needs to do. In practice, this doesn’t seem necessary, as we will see.

² – If we represent the edge destination by a 32-bit integer and the update value by a single-precision float, we’d end up with each machine sending and receiving ~15 GB of messages per iteration when processing the widely-used twitter_rv graph. Clearly not a good plan.

³ – About 3 GB per iteration for each machine on the twitter_rv graph.

⁴ – The CamSaS cluster consists of 16 machines, each equipped with an Intel Xeon E5-2430Lv2 CPU (12 hyperthreads, 2.4 GHz), 64 GB DDR3-1600 RAM, a Micron P400m-MTF SSD for the root file system and a Toshiba MG03ACA1 7,200rpm harddrive for the data, and both an onboard 1G NIC and an Intel X520 10G NIC. We run the default Ubuntu 14.04 (trusty) Linux kernel 3.13.0-24, and the harddisk is partitioned using the ext4 file system. We use the standard Spark 1.4 distribution, which includes GraphX.

⁵ – We suspect that our results are slightly better than those in the GraphX paper because we use dedicated machines (as opposed to m2.4xlarge EC2 instances).

⁶ – You might wonder why this single-threaded measurement is slower than the simple single-threaded one from before: it’s because structuring the program as dataflow adds some overhead.

⁷ – The smarter single-threaded implementation uses an “exotic” graph layout (based on a Hilbert space-filling curve); although this technique could be applied to timely dataflow, we won’t use it in the distributed implementation in this post.

Differential graph computation

Tue, 12 May 2015 15:20:00 +0000

I recently received an email asking if differential dataflow might be a good fit for breadth-first search in evolving graphs. I think it is a great fit, and I pushed a breadth-first search example to the differential dataflow repository in my delight and satisfaction.

Around the same time I got a different email, asking if the recent CellIQ paper from NSDI, which does incremental connected components, is fundamentally new or might their “GStream” API be easily cast in something like timely dataflow or differential dataflow.

Now, I suspect this person was just a disgruntled trouble-maker. The GStream API (Listing 2) is clearly covered by differential dataflow, which allows arbitrary incremental updates to iterative computations containing operators like join and group_by, enough for many data-parallel graph computations. Whether it can be done is different from whether it can be done well, though.

The CellIQ paper is one step ahead of us, and observes that although graph computation can be expressed “using join and group_by, this can be very inefficient”, because “join operators are not optimized for graph processing and can be very expensive.” They then cite some papers which, perhaps coincidentally, try to sell you their special-purpose graph processors.

These authors are not wrong: it can be very inefficient and expensive, especially if you use a previous-generation platform not designed from the ground up to handle streaming iterative computation.

Let’s see what happens if you build on something better, using just join and group_by, of course.

Differential dataflow

Differential dataflow is a way of implementing data-parallel dataflow computations so that work is done only in response to changes in the data. If you change one input to your computation, you would prefer not to re-evaluate the entire computation. Rather, changes to the input produces some changes in the output of your first dataflow operator, which are then changes to the inputs of subsequent operators. Eventually these changes either evaporate or spill out the end of your dataflow as changed outputs.

Chasing these changes and doing only this work is what differential dataflow does well.

Importantly—and as far as we are aware, uniquely—differential dataflow also maintains this property in the context of iterative dataflow computations.

Going through the math for this usually results in long-lasting head-aches for the readers, so I’ll summarize instead. As changes flow around the dataflow, they “happen” at various logical times. These times reflect the epoch of streaming input data, and iteration counters for any loops they may be contained in. Rather than eagerly aggregate these changes, we leave them disaggregated, allowing future computation to use arbitrary subsets of the changes. Although we leave ourselves more work to do for each change, we end up with an orders-of-magnitude reduction in the numbers of changes.

We might be getting ahead of ourselves with detail. Let’s step back and set some context.

Data-parallel dataflow

The big-data-scenti have caught up with the idea that we should use high-level programming constructs to describe data-parallel computations. Two operators I’ll detail are group_by and join.

Warning : I’m going to over-simplify some of the operators in the coming sections. I recommend checking out the code for the real details, some of which are smart and some of which are gross.

Group By

The group_by operator acts on a collection of records. I’ll imagine a type Stream<D> to indicate a not-yet-fully-formed data collection of records of type D. In this setting, group_by has the following sort of signature:

fn group_by<K, V, F, R>(inp: Stream<(K,V)>, red: F) -> Stream<R>
    where F: Fn(&K, &[V], &mut Vec<R>);

For those of you familar with map and reduce, the group_by function is basically like reduce. The operator acts on a stream of key-value pairs, groups incoming pairs by their key value, and applies the reduction function red to each non-empty pair of key and matching records.

They way I’ve done this, it gets a mutable vector of elements R and it is supposed to write into them. I’m open to other suggestions. It turns out that writing tasteful interfaces to performant code is hard.

Join

The second important (for this post) data-parallel operator is join, a binary operator matching pairs of input records with the same key.

fn join<K, V1, V2>(in1: Stream<(K,V1)>, in2: Stream<(K,V2)>) -> Stream<(K,V1,V2)>;

The join operator is bi-linear, which means it has some nice mathematical properties (roughly: each input can be processed record-at-a-time, oblivious to other records from the same input).

Other operators

So I lied. We are going to add some more simple operators such as map, filter, and concat. These operators do not involve any data movement or grouping, and can always be performed in situ. They are also linear, and consequently do not need any special differential dataflow implementation.

Prelude: Bread-first search

Let’s use what we have so far start to build up an example of breadth-first search.

Our approach will be to maintain a stream dists of pairs (node, dist), indicating that node can be reached using dist edges. From a supplied set of edges edges, we want to expand this set by one hop: each node provides the distance in which it can be reached to each of its neighbors, who then retain the smallest distance they’ve seen. I’m going to use u32s for node identifiers. Sorry.

Our code follows this intuition, by using a join to provide the per-node information to neighbors. Under the covers, join maintains a list of key-value pairs as a trie: a flat list of keys, each with an offset into a flat list of values. Ignoring than the fact that these lists might change, this is the same as the adjacency list representation of a graph.

The output of join then needs to go through a map to increase the proposed distances by one.

To the proposals produced by join we concat the original distances. You might want one of those.

Finally, we use group_by to collect the distance proposals. Here I’m going to cheat and use the fact that my implementation of differential dataflow sorts records in each group (an important part of “consolidation”; details later). To find the least distance, we can just pop the first distance in the sorted order; we are guaranteed one exists, because we wouldn’t have been called if there was no data.

fn step(dists: &Stream<(u32, u32)>,
        edges: &Stream<(u32, u32)>) -> Stream<(u32, u32, u32)> {

    dists.join_u(&edges)
         .map(|(_, l, d)| (d, l+1))
         .concat(&dists)
         .group_by_u(|_, s, t| t.push(s[0]))
 }

You might have noticed that join_u and group_by_u each have a _u suffix. Rust doesn’t think highly of operator overloading, and so you need to have distinct method names if you would like to have distinct implementations. In this case, we are using an implementation that knows we have unsigned integer keys, and can use a more efficient index (a Vec rather than a HashMap).

Iterative data-parallel dataflow

We need one more operator. A different kind of operator. A higher-order operator: we need iterate.

The iterate operator has a simple signature (these higher-order things always do, don’t they?)

fn iterate<D, F: FnOnce(Stream<D>)->Stream<D>>(s: Stream<D>, f: F) -> Stream<D>;

The operator takes a stream, and a description of how to act on the stream. Its output is the result of acting on the input stream in the supplied manner a very large number of times.

Of course, since iterate just takes a description of what to do repeatedly, it doesn’t define any data movement or new computation itself. In actual fact, we will need a few helpers behind the scenes to tell just when we can stop computing, but we don’t need to detail them to understand what iterate is supposed to produce.

The iterate operator is incredibly important for efficient computation. It explains to your computer that you plan to do the same thing multiple times, and that it might be a good idea to spend a bit of effort to take advantage of that. Although you can do graph computation by repeatedly invoking join and group_by, you are doing your computer a disservice if you treat the invocations as unrelated.

Breadth first search

The combination of iterate with our step function produces a breadth-first search implementation.

The only non-obvious step is that the edges stream needs to be brought in to the body of iterate. This is not easy to do automatically (at least, not at the moment), and so we explicitly do this with the help of a stream that is inside the iterative context: dists.

// iteratively expand distances
roots.iterate(|dists| {
    step(&dists, &dists.context().enter(&edges))
}

In the fullness of time it may make sense for iterate to take a function of two parameters, a stream like dists and some sort of context object that does the things we had to ask dists to do for us.

That doesn’t really look all that complicated, does it? Now, we don’t yet know very much about what differential dataflow is doing under the covers, but that doesn’t have to stop us from grabbing the code and running it.

ALL SYSTEMS GO!!!

I turned this on with 500M edges and 100M nodes, using nodes 0, 1, and 2 as roots (so the distances are the distances from any of these three nodes). I then asked it to update (randomly re-wire) one edge per second, so that I could see how long updates take. If you do the same thing, you’ll see some output that looks like:

00s: observed at (Root, 520): 2 changes; elapsed: 0.004067797010065988s
00s: observed at (Root, 522): 2 changes; elapsed: 0.003325369005324319s
00s: observed at (Root, 524): 2 changes; elapsed: 0.002495013002771884s
00s: observed at (Root, 526): 2 changes; elapsed: 0.0021344090055208653s
00s: observed at (Root, 527): 2 changes; elapsed: 0.00011885000276379287s
00s: observed at (Root, 532): 2 changes; elapsed: 0.0019779420108534396s
00s: observed at (Root, 541): 2 changes; elapsed: 0.0032094839989440516s
00s: observed at (Root, 545): 3 changes; elapsed: 0.0020695959974545985s
00s: observed at (Root, 548): 2 changes; elapsed: 0.0017733170097926632s
00s: observed at (Root, 553): 2 changes; elapsed: 0.001985581999178976s
00s: observed at (Root, 577): 1 changes; elapsed: 0.0019018270104425028s

This tells us a few things: first, it takes about 500s to get the computation up and running. That is a lot of time to do BFS. We are spending time laying out the graph in a nice fashion, getting reading for incremental updates, all that stuff. I’m not going to try and defend this as good performance.

Did I mention this was single-threaded? It goes faster with more cores and computers. I’m pretty sure.

Once we start going we see that each incremental update (one random edge removal and addition) happens pretty quickly (the numbers over on the right). Most of these are single-digit milliseconds.

You might also notice that we miss some epochs; the node distances don’t always change, and in those cases we don’t print anything. They also go fast, but we don’t want to pat ourselves on the back just because the data didn’t change.

Graph connectivity

Way back when, someone said doing graph computation using join and group_by was inefficient.

Let’s return to that, shall we?

Graph connectivity is usually done with an algorithm very similar to BFS. Rather than propagate a distance that we increase by one, we propagate a set of labels (the same set as the node identifiers), and each node keeps the smallest label it sees. The only difference from BFS is that we start each node with its own label, rather than a few roots with distance zero (and we don’t add one to anything).

It turns out that this algorithm is horrible. We should all stop using it. Even graph-processing experts.

The algorithm is dumb because we start circulating all labels, even very large labels that will eventually get dominated by smaller labels. This is a waste of everyone’s time.

Improving with prioritization

A better algorithm, from at least as far back as the PrIter paper, is to start circulating small labels first, and move on to larger labels only once nodes have gotten a chance to see if they should bother proposing their name as a label. If they see a better label in the meantime, they don’t.

You can do this in differential dataflow with a fancier version of enter, that method that brought our set of edges in to the loop. We have an enter_at method that lets you specify a function from record to iteration number, and the record first gets added to resulting collection at that iteration number. This lets us write clever things like:

let nodes = inner.context().enter_at(&nodes,
    |r| 256 * (64 - r.as_usize().leading_zeros() as u32));

What a mess. What this is saying is that we should introduce a record (here a node identifier) at an iteration equal to 256 times its base 2 logarithm. Why that number? It turns out it is a (provably) good trade-off between giving labels enough time to propagate and not using too many distinct batches. Each batch gets a 256 iteration head-start on the next batch, which grow in size geometrically.

This approach is a bit different from PrIter’s approach, which just says “start running and always use the smaller labels”. This works great in PrIter’s more ad-hoc setting, but we really need to be clear about which iteration is which for differential dataflow to work its magic.

Running things

Since graph connectivity is a bit different from BFS, in particular we have to symmetrize the graph before doing the propagation, we are going to use a different number of nodes and edges than above. I chose 20M nodes and 400M edges, generated randomly, because those numbers are pretty similar (in magnitude) to taking half of the vertices in the twitter_rv dataset (about 42M nodes and 1.5B edges).

It takes about 800s to compute the connectivity information. This is slower than fancy graph processing systems on the twitter_rv dataset, and we are only processing one fourth the data. The slowness can be partially explained by the fact that we are generating a bunch of random data, ingesting and sorting it, paging because reasons, and then doing graph connectivity using differential dataflow, which does math and stuff. Also, I just wrote this stuff last week and haven’t found a good profiler yet.

It is also worth pointing out that the 800s time is one thread, and the computation is inherently parallel. If I had a cluster I could tell you how well it parallelizes. Given that I don’t have a cluster (or a profiler) it might take a bit of love before it does, but it has been done before.

Anyhow, it would be interesting to see if it has similar performance to the graph processing systems, because it can totally do some things they can’t even approach.

Tell us what can it do, Frank!

So we have our 400M edge graph, and we want to change it. Maybe someone tweetled an instachat.

We’ll do this two ways, to get a good latency measurement and then a good throughput measurement. Specifically, let’s try changing one edge at a time, and then let’s try changing 1,000 edges at a time.

We get the following cumulative density functions for the response time (from change introduced until confirmation that all output changes produced) for randomly tweaking one and one thousand edges, respectively:

To explain, if we do one update at a time we get a median response time of 130 microseconds. If we do 1,000 updates at a time we get a median response time of 33 milliseconds. At this rate we should be able to do 30,000 edge updates per second, observing the correct graph connectivity output with a faster refresh rate than your favorite feature films.

The elapsed running times for graph connectivity are within striking distance of purpose-built graph processors running on clusters (stay tuned!), and we get near real-time interactive updates too boot.

Hey, and we did this all with just join and group_by (and iterate, to be fair). Who likes apples?

Up next: Under the covers

I’m going to try and explain differential dataflow again. It is pretty exhausting, but I think it is the sort of thing people need to understand if they want to work in the space of real-time graph analytics. I’ve got a bit of travel in the coming weeks, so in the meantime if you are keen you can totally check out the repo and lament the lack of intelligent comments and tests.

I’m hard at work tweaking the performance (mostly memory footprint) of differential dataflow. There are still a few cool things to do, and if you are interested in getting involved, let me know!

Abomonation: terrifying serialization

Mon, 04 May 2015 15:20:00 +0000

Today we’re going to look at a simple serialization library written in Rust. Simple and utterly terrifying.

The library, Abomonation (typo intentional), uses Rust’s unsafe keyword in some interesting ways. The intent is that it is actually safe if you only deserialize binary data serialized by the library.

I should say that many of the things going on in here look a lot like what goes on in CapnProto. If you are serious about being a grown-up about things, I would absolutely look over there. Among other things, it has the advantage of not necessarily lowering your opinion of me, unlike this post.

Serialization and Deserialization

Our goal is to write a pair of methods with the following signatures:

fn encode<T>(typed: &[T], bytes: &mut Vec<u8>);
fn decode<T>(bytes: &[u8]) -> &[T];

That is, if you present a slice of data &[T] we can populate a Vec<u8> with appropriate binary data. Similarly, if you provide us some binary data &[u8], we can interpret it as typed data &[T] for you.

We aren’t going to do this by the end of the post, but we’ll have something pretty similar. These are the actual signatures at the moment (the implementations are at the end of the post; don’t read them yet!):

fn encode<T: Abomonation>(typed: &Vec<T>, bytes: &mut Vec<u8>);
fn decode<T: Abomonation>(bytes: &mut [u8]) -> &Vec<T>;

There are important distinctions here, so let’s briefly think about what’s being said by the interfaces.

We are only going to be able to do this for some types T, those implementing Abomonation. This makes sense, because some types are hard to serialize. The name doesn’t make sense. Yet.
We are taking a &mut [u8] rather than a &[u8]. That’s weird. Why would we need exclusive access to the input data? Are we going to change the binary data we’ve received? (I’m so sorry.)
We are returning a reference to a Vec<T>. Where did that Vec<T> come from and who owns it? The decode function doesn’t take one as a parameter, and you (usually) don’t just get references to owned data out of nowhere.

Well isn’t that just a big pile of mysterious language? Apparently we are going to do some things, and you should all be terrified.

Encoding and Decoding nice data

Let’s start with some easy examples, ones that use the unsafe keyword but which we’ll pretty easily convince ourselves are really pretty friendly. We’ll get to learn a bit about how Rust manages its data under the covers, and maybe get our feet wet. What could go wrong?

The following function takes a slice of typed data and presents it back as a slice of binary data.

unsafe fn typed_to_bytes<T>(slice: &[T]) -> &[u8] {
    std::slice::from_raw_parts(slice.as_ptr() as *const u8,
                               slice.len() * mem::size_of::<T>())
}

Notice the unsafe label on the function. This is because it uses the from_raw_parts method, which is itself unsafe (it doesn’t check to make sure that the underlying data look correct). In this case it is safe, because we only want to look at the binary data (not mutate it) and every byte is a valid u8.

I could choose to remove the unsafe label and put unsafe interally like so, declaring the method safe:

fn typed_to_bytes<T>(slice: &[T]) -> &[u8] {
    unsafe {
        std::slice::from_raw_parts(slice.as_ptr() as *const u8,
                                   slice.len() * mem::size_of::<T>())
    }
}

I have been warned this is bad, though, because reading a struct-padding byte is undefined behavior. This will not be the most irresponsible thing that happens in this post.

On the other end of the safety spectrum, I present:

unsafe fn bytes_to_typed<T>(slice: &mut [u8]) -> &mut [T] {
    std::slice::from_raw_parts_mut(slice.as_mut_ptr() as *mut T,
                                   slice.len() / mem::size_of::<T>())
}

This not only exposes mutable slices, but permits any type at all. Imagine what would happen if T was String, for example: String owns its [u8] buffer and cointains a pointer to it. You would be able to dereference arbitrary places in memory! Danger!! Danger!! The unsafe label stays on for this one.

Imagine you just saw bytes_to_typed eat a goat. It is powerful, but the unsafe fence protects us. Just like in that movie with all those cautionary tales that I didn’t stay for the end of.

You said “nice data”

These two methods work great if all you want to serialize are slices of primitive types. If you have a &[u64] you can just transmute it over to a &[u8] and write out the data. If you get a &[u8] back you can (in good conscience) know that transmuting it to &[u64] works well.

We are going to use the above two functions in our implementation of encode and decode, so let’s be brave and add the following sorts of implementations:

impl Abomonation for u8 { }
impl Abomonation for u16 { }
impl Abomonation for u32 { }
impl Abomonation for u64 { }
// lots of others ...

Notice that these implementations don’t do anything, they just indicate that it is safe for encode and decode to transmute them around. In fact, Abomonation will have a few methods, but these types can just use the default implementations.

The Abomonation stirs …

Primitive types are neat and all, but we are serious people! What about things like tuples? Surely I should be able to transmute a &[(u64, i8)] to binary and back, right? Yes, you should. But…

… it’s complicated. You see, we built this trait to reach beyond primitive types. Much further beyond.

Ok, let’s just write down this trait.

trait Abomonation {
    unsafe fn entomb(&self, _writer: &mut Vec<u8>) { }
    unsafe fn exhume(&mut self, _bytes: &mut &[u8]) { }
}

Wow, grim. In case unsafe wasn’t enough, I chose creepy method names. Creepy, but appropriate!

The two methods have the following intents:

entomb: having written &self as binary, serialize any further data it might own.
exhume: having populated &mut self with binary data, deserialize data it owned.

The reason this trait exists is because our method encode(&Vec<T>, &mut Vec<u8>) is just going to copy the contents of the typed vector into the binary vector. Seems harmless enough for the types implementing Abomonation so far, but we will use entomb to check with each T to see if they want to write any more. Similarly, decode uses exhume on each element to recover any owned data.

Wait, owned data? What? I mean, u64 doesn’t own any data, so does it need to do anything? Nope!
All those other primitive types don’t need to either. They can use the default empty implementation.

All those happy townsfolk, oblivious to what awaits…

We got this far because we needed to talk about tuples. Let’s see how we might implement the weirdly capable Abomonation for a pair of abomonable types.

impl<T1: Abomonation, T2: Abomonation> Abomonation for (T1, T2) {
    unsafe fn entomb(&self, writer: &mut Vec<u8>) {
        self.0.entomb(writer); self.1.entomb(writer);
    }
    unsafe fn exhume(&mut self, bytes: &mut &[u8]) {
        self.0.exhume(bytes); self.1.exhume(bytes)
    }
}

Really, the whole interface is probably just messsing with forces we don’t understand.

For example, you probably don’t understand why exhume uses a &[u8] when we said up above that we were going to use &mut [u8]. In retrospect, there were warning signs that something was wrong.

Late in the lab one night …

Some drinks in, an ambitious scientist finds she needs to serialize a &[Vec<Vec<u64>>]. She thinks:

Hey, couldn’t I implement Abomonation for Vec<T> where T: Abomonation?

Well, you can certainly type it.

impl<T> Abomonation for Vec<T> where T: Abomonation {
    unsafe fn entomb(&self, writer: &mut Vec<u8>) {
        // write out the contents of self
    }
    unsafe fn exhume(&mut self, bytes: &mut &[u8]) {
        // read back the contents into &mut self
    }
}

You know, this might just work. A Vec is a pointer, a length, and a capacity. Three usize values. Serializing this triple doesn’t serialize the Vec’s contents, but we can just do that in Vec’s entomb. Similarly, when we pick up the serialized form of the Vec, a triple, it doesn’t point at anything valid (to our great shame, it points at some location in memory wherever we serialized this), but exhume gives us a chance to read valid data back and make things right.

If the scientist did something responsible in exhume, like create a new Vec<T> with new backing memory and assign it to *self, the story ends happily. But our story doesn’t follow that path.

Writing the code that shouldn’t be written

The scientist, unfortunately, was mad with power. She demanded zero allocations. She would use the binary data as provided to back her creepy, hard-to-reason-about deserialization machinations.

Rust provides unsafe functions capable of assembling a Vec out of “raw parts”: a pointer, a length, and a capacity. Our scientist is going to connect wires not meant to be connected, by building a Vec using a pointer into the supplied byte array.

Now, our scientist may be power-mad, but she is still a scientist so she’s going to do this properly.

unsafe fn exhume(&mut self, bytes: &mut &[u8]) {

    // 1. extract the memory from bytes to back our vector
    let binary_len = self.len() * mem::size_of::<T>()
    let buffer = &bytes[..binary_len];
    *bytes = &bytes[binary_len..];

    // 2. transmute buffer to &mut [u8], and then to a &mut [T].
    let slice: &mut [T] = bytes_to_typed(mem::transmute(buffer));

    // 3. build a new vector using this memory
    let vector = Vec::from_raw_parts(slice.as_mut_ptr(), self.len(), self.len());

    // 4. overwrite *self w/o dropping it.
    std::ptr::write(self, vector);

    // 5. pretend everything is normal; call exhume on each element
    for element in self.iter_mut() { element.exhume(bytes); }
}

Pretty much all of these lines (everything except step 1.) are unsafe. Because the method has the unsafe tag, we don’t need to flag individual regions as unsafe. They are all quite unsafe.

Let’s take a tranquil moment to reflect on what’s been done, and what might happen. We’ve minted a Vec<T> pointing at memory that cannot be freed. The memory is valid, and the rest of the Vec<T> appears legitimate (length, capacity). But, as soon as we return from this method that Vec<T> is loose in the country-side and who knows what might happen.

Well, what could go wrong, really?

GRAAAAAAAAARRRRR!!

You know, actually it all sort of works.

GRAAAaaaoooo?

Yeah, there is a disasterous mess of things here, and really Rust’s memory safety isn’t touching any of this with a ten foot lifetime bound. But let’s talk about the possible issues.

First, let’s be clear that exhume is unsafe. Don’t call it. I haven’t figured out how to keep people from calling it yet (public traits export all their methods), but just don’t. It makes a few assumptions about how it will be called. It assumes that:

The &[u8] data is, in fact, exclusively held. We go and transmute it to &mut [u8] which is very wrong unless we are sure that we have the only reference. Fortunately, decode takes a &mut [u8] as its argument, and this ensures that our references are exclusive. We are also careful to partition the &[u8] into disjoint parts when we use it, to maintain this invariant.
The &mut self parameter to exhume will only be presented outwards as a &self. We can’t let anyone mutate or own this data. Trying to add elements to Vec will trigger a re-alloction, which would attempt to release the memory. It is very important that all references to the data are just that, only references. And references with lifetimes bounded by that of the source &[u8].
The &mut self parameter to exhume will never be dropped. Dropping the Vec would attempt to release the backing memory, which … well it isn’t going to work out. Fortunately, there never was a Vec. There were some bytes, and we pretended that a reference to the bytes was actually a reference to a Vec, but there was never a Vec to drop.

There are probably some other assumptions. Everything breaks if you pass in invalid data, for example. Don’t do that either.

In fact, sorting out whether these assumptions are properly nailed down, or whether something truly horrible has happened, is basically what I’m up to now. I don’t really know, and it isn’t particularly clear what I need to do to use unsafe correctly. I invite your thoughts and criticism.

Rust’s position is, perhaps refreshingly honestly: “yeah, it’s unsafe. you even said so.”

Epilogue

What about our ambitious, power-mad scientist; where did she get off to?

While we have been fretting about what might have gone wrong, she is experiencing excellent throughput numbers. Let’s look at a few using Rust’s benchmarking facilities.

Encoding

Here is a helpful routine to spin Rust’s Bencher struct on any type implementing Abomonation.

fn _bench_enc<T: Abomonation>(bencher: &mut Bencher, vector: &Vec<T>) {

    let mut bytes = Vec::new();
    encode(vector, &mut bytes);
    bencher.bytes = bytes.len() as u64;
    bencher.iter(|| {
        bytes.clear();
        encode(vector, &mut bytes);
    });
}

We try it out with types Vec<u64>, Vec<String>, and Vec<Vec<(u64, String)>> and see:

test bench_enc_u64     ... bench:       411 ns/iter (+/- 84) = 19990 MB/s
test bench_enc_string  ... bench:     12039 ns/iter (+/- 3330) = 2893 MB/s
test bench_enc_vec_u_s ... bench:     12578 ns/iter (+/- 1665) = 3482 MB/s

The String is just format!("grawwwwrr!"). Results may vary with the string’s length, silliness.

I should also point out that the throughput is not goodput. It includes the extra pointers and capacities and stuff we didn’t need to send. The numbers do get a bit worse, especially for short strings. It’s the sort of thing you could fix by having a buffer implement encode/decode and push/pull just lengths. However, you do need to stage the Vecs somewhere, and our approach let bytes do that for us.

Decoding

What about deserialization? That is where all the mess is, right? Well, it is a bit hard to measure deserialization speed fairly, because we are really not doing much other than fixing up some pointers.

I did put some testing code in place to make sure that we were getting the right results back:

fn _bench_dec<T: Abomonation+Eq>(bencher: &mut Bencher, vector: &Vec<T>) {

    let mut bytes = Vec::new();
    encode(vector, &mut bytes);
    bencher.bytes = bytes.len() as u64;
    bencher.iter(|| {
        let result = decode::<T>(&mut bytes[..]);
        assert!(result.len() == vector.len());
        for i in 0..result.len() {
            assert!(result[i] == vector[i]);
        }
    });
}

These give pretty comparable results. Apparently pushing a bunch of bytes works at about the same rate as checking that those bytes are what you expected. Again, not goodput numbers; caveat emptor.

test bench_dec_u64     ... bench:       525 ns/iter (+/- 262) = 15649 MB/s
test bench_dec_string  ... bench:     11289 ns/iter (+/- 2432) = 3086 MB/s
test bench_dec_vec_u_s ... bench:     12557 ns/iter (+/- 2183) = 3488 MB/s

Just to show how silly things are, here are the numbers where we comment out the assert loop, just checking the resulting lengths.

test bench_dec_u64     ... bench:         2 ns/iter (+/- 0) = 4108000 MB/s
test bench_dec_string  ... bench:      2625 ns/iter (+/- 1031) = 13272 MB/s
test bench_dec_vec_u_s ... bench:      3020 ns/iter (+/- 1266) = 14503 MB/s

Hey look, 4TB/s deserialization. Maybe it is time to head back to Silicon Valley, maybe start things up.

Appendix: encode and decode

I wanted to put the implementations up here, but they aren’t very pretty. In particular, I suspect they could be greatly improved (possibly by deletion).

pub fn encode<T: Abomonation>(typed: &Vec<T>, bytes: &mut Vec<u8>) {
    let slice = unsafe { std::slice::from_raw_parts(mem::transmute(typed), size_of::<Vec<T>>()) };
    bytes.write_all(slice).unwrap();
    unsafe { typed.entomb(bytes); }
}

pub fn decode<T: Abomonation>(bytes: &mut [u8]) -> &Vec<T> {
    let (split1, split2) = bytes.split_at_mut(mem::size_of::<Vec<T>>());
    let result: &mut Vec<T> = unsafe { mem::transmute(split1.get_unchecked_mut(0)) };
    unsafe { result.exhume(&mut &*split2); }
    result
}

The whole pile of stuff will be up on github and crates.io eventually, but I wanted to take at least a bit of time with other eyeballs on this to see if a) it is all horribly wrong, and b) whether the risk is ever worth it.

Also, I should obviously avoid serializing pointers to your memory locations. No one needs to see that.

Data-parallelism in timely dataflow

Sun, 19 Apr 2015 15:20:00 +0000

The previous post described a neat algorithm of Ngo et al and then described how one could go and implement this in timely dataflow. There were several bits of Rust code, a few places where details were glossed over, and then the claim that this was now a data-parallel implementation of the neat algorithm.

In this post we walk through how the data-parallel bit happens, and detail some of the moving parts.

Let’s start at the top, with some code we didn’t see in the previous post: a main method:

fn main() {
    let workers = 4; // whatever you like here!
    let mut guards = Vec::new();
    for communicator in ProcessCommunicator::new_vector(workers) {
        guards.push(thread::Builder::new()
                        .name(format!("worker thread {}", communicator.index()))
                        .scoped(move || triangles(communicator))
                        .unwrap());
    }
}

Here we identify some desired number of workers, ask ProcessCommunicator (whoever that is) to create a new vector (of something), and for each element of this vector we start up a thread. Now, the elements of that vector have been suggestively named communicator, and indeed they each implementation a Communicator trait. The scoped method is where the thread is started, and as you can see the only input it takes is one of these communicators. What could they possibly do?

Communicator

The Communicator trait provides the only point of contact between workers in timely dataflow. It has a fairly simple definition, so let’s look at it:

pub trait Communicator {
    fn index(&self) -> u64;
    fn peers(&self) -> u64;
    fn new_channel<T:Send+Columnar+Any>(&mut self)
        -> (Vec<Box<Pushable<T>>>, Box<Pullable<T>>);
}

The only functionality a communicator exposes to workers is to tell them their identity, the number of other workers, and to set up channels with the other workers. The new_channel method returns some targets the worker can push typed data at (one per worker) and one source the worker can pull from.

When a worker pulls from a source they receive data pushed by other workers into their corresponding targets. The workers are expected to create the same sequence of channels; things go sideways if not.

That’s all. If workers want to communicate in any other ways, they’ll need to create some channels.

Using communicators

Let’s look at the last bit of code from the previous post, the triangles method referenced just above, in which we define the dataflow computation and set it going. I’m going to base this off of the code as it currently exists in the repository, which has a few syntactic differences from the previous post.

fn triangles<C: Communicator>(communicator: C) {

    // load up our fragment of the graph
    let graph = Rc::new(RefCell::new( /* load up mmap'd file */ ));

    // define extenders for a -> b and  (a, b) -> c, respectively
    let b_extend = vec![&graph.extend_using(|| { |&a| a as u64 } )];
    let c_extend = vec![&graph.extend_using(|| { |&(a,_)| a as u64 }),
                        &graph.extend_using(|| { |&(_,b)| b as u64 })]

    // create a new root dataflow context
    let mut root = GraphRoot::new(communicator);
    let mut input = {
        let mut builder = root.new_subgraph();
        let (input, stream) = builder.new_input();

        // enable, extend, extend
        stream.enable(builder)
              .extend(b_extend)
              .extend(c_extend);

        input  // return the input handle
    };

    // iterate until done
    while root.step() {
        // introduce input
    }
}

Perhaps the most important thing to observe about this method is that it is per-communicator logic. Each worker will call this method independently, and so they need to make sure they act in concert. This mostly boils down to constructing channels in some consistent order, and not going off and doing something else while other workers are waiting for you.

Ideally the code isn’t too threatening. The first few lines are related to the previous post, where we load up some graphs and define “prefix extenders” used to drive the aforementioned neat algorithm.

There are some very important lines, though, so let’s look a bit more carefully at them.

Creating a dataflow graph root

The point at which the dataflow graph first comes in to existence is when we wrap up our trusty communicator in a GraphRoot. This is that momentous line of code:

let mut root = GraphRoot::new(communicator);

A GraphRoot is pretty simple, actually. It is the simplest implementator of the GraphBuilder trait, whose main role in life is to hand back references to a Communicator so that we can make channels.

There are much more advanced and interesting, implementors of GraphBuilder. Let’s see one now!

Creating a dataflow subgraph

The next fragment of code creates a dataflow subgraph.

let mut subgraph = root.new_subgraph();

The new_subgraph() method, defined by the GraphBuilder trait, returns a SubgraphBuilder. This is that advanced and interesting implementor of GraphBuilder I mentioned above.

The SubgraphBuilder provides access to a Communicator and the ability to call new_subgraph, but it also has a non-trivial implementation of the method add_scope<S: Scope>(scope: S).

Of course, I’m sure you all remember what a Scope is, right? (\(\leftarrow\) click! do it. omg… click! click!).

A refresher on Scopes

The Scope trait defines the methods different components of a timely dataflow graph need to use to discuss the progress of execution in the graph. A scope can explain to its parent how many inputs and outputs it has, what sorts of messages it might plan to send on those outputs, and given the chance maybe do a little bit of work to make progress itself, which it can then report to its parent.

There are more details in the link above, and much more to say about the progress tracking protocol.

Our first example of a scope is actually in the very next line of code:

let (input, stream) = subgraph.new_input();

The input method is provided by an extension trait on implementors of GraphBuilder. Let’s see it.

pub trait InputExtensionTrait<G: GraphBuilder> {
    fn new_input<D:Data>(&mut self) -> (InputHelper<G::Timestamp, D>,
                                        Stream<G::Timestamp, D>);
}

Ok, that might have been too much information. We see that new_input does return two things, and apparently these are an InputHelper and Stream, but what these are is presently a total mystery. Also, what does all that G::Timestamp noise mean?

Last things first, G::Timestamp is an associated type of the graph builder G. Each graph builder has an associated timestamp, and all scopes that graph builder will tolerate must use that timestamp. Both InputHelper and Stream are defined in terms of a specific common timestamp, given by G. They each also have a second type parameter, D, which is the type of data the input will pass along.

The implementation of input is neither particularly self-explanatory or illuminating, so I’ll summarize. The method does a few things, including creating an InputHelper (to push data at) and a Stream (to connect dataflow to). Importantly, it also creates an InputScope, which implements Scope and is what makes promises about which input timestamps are sealed. It is added to the subgraph builder.

The implementation of input also snags a channel from the communicator, and wires it up so that submitted data get routed directly to consumers of Stream. Other than keeping track of the number of records sent, which InputScope needs, all of the logic lives behind the channel implementations.

Intermission: data-parallelism

Let’s take a moment to reflect on where we are. We’ve defined some infrastructure that will let each worker send records in to a dataflow computation, and, if we keep it up, exchange these records with other workers. Although a lot of our discussion here sounds like we are just talking about a single thread of execution, that is exactly how we want it.

One of the main virtues of data-parallel programming is that we get to define our computation as if it were single threaded, knowing that if we spin up an arbitrary number of copies of the workers, and exchange data between them in an appropriate manner, the same result comes out the other end.

This programming pattern does come with restrictions, mainly that all communication must be explicit and that the structure of communication (the dataflow graph itself) is defined ahead of time, but subject to these restrictions we don’t have to think too hard about concurrency. That is really good news.

Adding some more scopes

Let’s get back to work. Our worker must define what it should do with this input stream now that it has it.

At this point we get in to a slightly mysterious detail of our current implementation, which connects to Rust’s seriousness about shared access to mutable state. Rust is really quite serious about this topic.

The graph builder builder owns a bunch of mutable state, and we need to be very clear about when we are using it and when we are not using it. We we explicitly asked builder to create an input for us, which is pretty clear. At the same time, we’d probably rather not involve a reference to builder in each method call, but we must know who is currently responsible for it.

What we have at the moment is a notion of an ActiveStream, which is the information about a stream (basically a name and a listener list one can join) plus a graph builder. The ActiveStream owns that graph builder, in the Rust sense, so it knows that it has exclusive mutable access to its services.

We create active streams from inactive streams by calling enable with a graph builder argument.

stream.enable(builder)
      .extend(b_extend)
      .extend(c_extend);

You might look at this and say: “whoa, is builder gone now?” In this case, yes. However, you can always get it back from an active stream; we just happened to discard the results in this case.

Additionally, and this was someone else’s clever idea, not mine: there is a blanket implementation

impl<G: GraphBuilder> GraphBuilder for &mut G {
    // lots of "fn func(&mut self) { (**self).func() }"
}

which says we can use a mutable reference &mut G anywhere we would use a G: GraphBuilder. So we could have replaced builder up there with &mut builder, and not risked losing track of it.

Actually adding more scopes

Writing extend a few times is sort of a cop-out as an example of additional scopes. Not so helpful.

In the previous post I elided the implementation of StreamPrefixExtender, the trait providing actions on ActiveStream including things like counting the number of extensions, proposing extensions, and validating the extensions. Let’s go through one of these implementations.

I’m going to use propose as the simplest implementation. The other two are roughly the same, but with slightly more inner logic. The propose implementation uses an extension method unary that I’ve written, wrapping the implementation of a scope with one input and one output. We will move to its implementation in just a moment, but let’s see its arguments to understand what it needs to know.

impl<P, E, G, PE> StreamPrefixExtender<G, P, E> for Rc<RefCell<PE>>
where P: Data+Columnar,
      E: Data+Columnar,
      G: GraphBuilder,
      PE: PrefixExtender<P, E>+'static {
    fn propose(&self, stream: ActiveStream<G, P>) -> ActiveStream<G, (P, Vec<E>)> {
        let clone = self.clone();
        let exch = Exchange::new(|x| hash(x)); // <-- minor lie
        stream.unary(exch, format!("Propose"), move |handle| {
            let extender = clone.borrow();
            while let Some((time, data)) = handle.input.pull() {
                let mut session = handle.output.session(&time);
                for datum in data {  // send each extension
                    session.give(p, extender.propose(&p));
                }
            }
        })
    }

The unary method takes three arguments: a description of how its inputs need to be exchanged (to make sure the data arrive at the right worker), a String naming the operator, and logic that takes a handle through which inputs can be read and outputs sent.

If we think for a moment, there really isn’t all that much more that needs to be said about an operator:*

Which records should go to which worker.
What should the worker do with the inputs when they arrive.

In the example above, we just want to consistently route prefixes to the same worker, so we use a hash of the prefix. When presented with some prefixes, the right thing to do is extend each of them and send the pair of prefix and extensions. Nothing else to say about the logic here, really.

*: Actually there is more to know about, namely notification, which is a hugely important part of timely dataflow, but we just haven’t gotten there yet. Subsequent post, for sure.

Unary scope

The unary method hides a lot of boiler plate, so that we can focus on the dataflow operator logic.

What actually goes on inside that method? Nothing wildly complicated. It is about 90 lines of code. When unary is called it uses the first argument, exch above, to transform a channel from the communicator into a simple push/pull interface. It registers the “push” part of this with stream and attaches the “pull” part to its handle object. It also prepares an ActiveStream for output, and connects the output of handle to the (to be populated) list of interested listeners. Finally, it creates a UnaryScope who owns handle and will both call the user logic on it when needed, and report the number of records received and sent to its parent scope.

That was a pretty long paragraph, so maybe it isn’t all that simple. But, it isn’t wildly complicated either.

Setting things in motion

The final bit of code looks like this:

while root.step() {
    // introduce input
}

That looks pretty simple and harmless, but it is what sets the whole dataflow computation into motion.

The root.step() call recursively traverses the scopes, asking each about any progress they’ve made. This is where each scope looks around, says “hey I do have input!”, does a little bit of work, and then tells its parent what its done. The return value indicates whether there remains more work to do.

In our case, the scopes will start moving input records, and start proposing extensions, and doing all the bits of logic we’ve programmed in. They will keep going as long as the inputs remain open, the operators report they are not yet done, and the progress tracking protocol reports unresolved work.

Crucially, all this business is written so that each of these workers operates independent of the others. While they do coordinate implicitly through data exchange and the progress tracking protocol, we just need to turn each of them on and run each of them until it stops.

Ideally this stopping happens sooner with more workers, but… well I don’t want to spoil the surprise.

Worst-case optimal joins, in dataflow

Sat, 11 Apr 2015 15:20:00 +0000

I’ve gotten timely dataflow in Rust up and running exciting computations! I’m going to explain one that I think is especially cool, and that I’m going to try using for some performance analysis of the underlying system (the system has so far been subjected only to latency micro-benchmarks …).

The code for everything I’ll talk about is available online. It isn’t particularly pretty yet, but just you wait.

Seriously, you might actually want to wait. In the meantime, you can read this excellent post!

Relational Joins

A relational join is a pretty well studied thing, and I’m just going to lay some bare bones description here so that we have some common terminology. The problem starts with a collection of relations, think tables; we’ll call these relations \(R_1, R_2, \ldots R_k\). There are also several attributes, which we will refer to as \(A_1, A_2, \ldots A_j\). Each relation names some subset of these attributes, and each element in the relation has a value for each named attribute. Not all relations need to use the same set of attributes.

The relational join problem is, given several relations, determine the set of tuples over the full attribute space so that for each tuple, its projection onto the attributes of each relation exists in that relation.

An example

Consider the first records of the three relations \(R_1, R_2, R_3\), over the three attributes \(A_1, A_2, A_3\):

The relational join between the three relations must contains at least the triples \((1, x, 2)\) and \((5, x, 4)\), because we can see in the first relation the pairs \((1,x)\) and \((5,x)\), in the second relation the pairs \((x,2)\) and \((x,4)\), and in the third relation the pairs \((1,2)\) and \((5,4)\).

Of course, there may be more records in the full join as we see more of the records from each relation.

Ye Olde Methodologees

There are many ways to go about doing a binary join, between two relations, but the simplest is a hash-join, where you look at the attributes in common between the two relations and hash each tuple based on their restriction to those attributes. For each pair of matching tuples (one from each relation), you form the extended tuple that takes the union of the attributes of the relations.

In the example above, we might join the first two relations, by hashing records using the attribute \(A_2\). This matches \((1,x)\) and \((5,x)\) from the first relation with \((x,2)\) and \((x,4)\) from the second relation. The output data in this case are the four triples: \((1,x,2)\), \((1,x,4)\), \((5,x,2)\), and \((5,x,4)\). The record \((1,q)\) matches nothing in the second relation, and results in no output.

To do a multi-way join, one could just keep grabbing relations and joining them in, until all relations have been used. This gives the correct answer, but can be really slow. A smarter way is to form a “plan”, which is a binary tree in which the leaves are relations and the internal nodes correspond to joins of relations. The root of this tree is the join of all relations, but the tree structure suggests which relations are good to start joining together.

In the example above, we might prefer to join the three relations by first joining \(R_2\) and \(R_3\), which would produce only two records before joining with \(R_1\). Building a smart join plan is something that database researchers like to talk about at their fancy meetings, and is a good way to strike up a conversation.

More recent work

Relational joins have been around for such a long while, you might be a bit surprised to learn that there is still new work going on here. You might be even more surprised to learn that, in some respects, folks have been doing it wrong for quite some time. This is exactly what Ngo et al observe, in awesome work:

The standard approach to computing relational joins, in which one repeatedly does binary joins, can do asymptotically more work than the join could ever possibly produce output tuples.

In the three-way join example, if each relation has size \(m\), there can be at most \(m^{3/2}\) output tuples, because math. However, inputs exist so that any plan based on binary joins will do do \(m^2\) work.
There exist algorithms that never do more work (asymptotically) than the join could, for some input of the same size, produce output tuples. For a three-way join, they will do \(O(m^{3/2})\) computation.

This second point doesn’t mean that they will only do as much work as they will produce output tuples, only that when they do lots of work they at least have the excuse that they might have had to do it.

You know who doesn’t even have that excuse? The standard approaches to computing relational joins.

Generic Join

The algorithm that Ngo et al detail is really quite general. They even call the algorithm GenericJoin.

I’m going to focus on a specific realization of it. I should say at this point that the specific realization is due to other people, not me. I’m not really sure who gets credit, but Semih Salihoglu and Chris Ré are the ones that taught me about worst-case join processing, and it was with Semih, Chris, and Michael Isard that we banged out the first version of this in Naiad.

Specific Join

Rather than think of adding in relations one at a time, the way classical join processing might, we are going to think about adding in attributes one at a time.

Imagine we had the join on attributes \(A_1, \ldots A_i\), and want to output the join on attributes \(A_1, \ldots A_{i+1}\). For each \((a_1, \ldots a_i)\) we want to produce the set of \((a_1, \ldots a_{i+1})\) supported by each relation \(R_j\).

The easiest way to do this is just to ask each \(R_j\): “what extensions \(a_{i+1}\) do you support for \((a_1, \ldots a_i)\)?” For each \((a_1, \ldots a_i)\) we intersect their results, and return \((a_1, \ldots a_{i+1})\) for each \(a_{i+1}\) in the intersection.

Of course, to ask the relations for the extensions and get the answer back quickly, each relation needs to be indexed by each prefix of attributes. This introduces redundancy, but we are going to work with it.

Smarter Join

Nothing about this algorithm is smart yet, but what Ngo et al observe is that if you do this intersection carefully, you get a worst-case optimal join algorithm. The specific care you have to take is, for each \((a_1, \ldots a_i)\), only do work proportional to the smallest set of candidate extensions from any relation \(R_j\). Rather than just intersect everything willy-nilly, we have to start with the smallest and work our way up.

Fortunately, computing the intersection of a small set with a large set is something that can take time roughly linear in the size of the smaller set. You can either hash everything all over the place, or eat a logarithmic factor (they are ok with that) using various binary search techniques.

But, let’s look for a second and make sure we understand this. For each \((a_1, \ldots a_i)\) we have to ask a data-dependent relation to propose some extensions, and then ask the other relations to validate them. There isn’t a static plan saying “everyone first ask \(R_1\), then \(R_7\), then ..”; rather than pipeline records through relations, like a traditional join plan might, we are going to exchange them all over the place.

“Exchange”, you say? I hope you can see where this is going.

A Rust implementation

Obviously we were going to do this. Don’t act surprised.

Some abstractions

Do we really want relations and tuples all over the place in our code? No! Let’s do some abstraction.

From the discussion above we can see that we really only need a few things from a relation:

It should be able to report how many extensions it would propose for \((a_1, \ldots a_i)\).
It should be able to propose specific extensions for \((a_1, \ldots a_i)\).
It should be able to intersect proposed extensions for \((a_1, \ldots a_i)\) with its extensions.

So let’s write a trait that does this. I’m going to call the tasks above “count”, “propose”, and “intersect”.

pub trait PrefixExtender<Prefix, Extension> {
    fn count(&self, &Prefix) -> u64;
    fn propose(&self, &Prefix) -> Vec<Extension>;
    fn intersect(&self, &Prefix, &mut Vec<Extension>);
}

I think this is as promised. This is what a relation needs to implement for us to be able to extend an element of type Prefix (think tuples \((a_1, \ldots a_i)\)) with an element of type Extension (think \(a_{i+1}\)).

Distributed implementation

Of course, what we would really like is to extend each of the prefixes in parallel across many workers. At least, that is what I want. If you couldn’t care less, you should totally just skim this part.

To make this happen I’m going to use the timely dataflow libary, which uses a Stream<G, Prefix> type to represent a distributed stream of records of type Prefix. The G type parameter describes how the stream is distributed and how the computation will execute, and we’ll just ignore it for now.

We need to lift the implementation of a PrefixExtender<P, E> to work on streams Stream<G, P>. Fortunately, I’m going to do this for us, by implementing the following trait for any type implementing PrefixExtender<P, E> (plus some information about how to distribute the prefixes among workers).

pub trait StreamPrefixExtender<G, P, E> {
    fn count(&self, &mut Stream<G, (P, u64, u64)>, u64) -> Stream<G, (P, u64, u64)>;
    fn propose(&self, &mut Stream<G, P>) -> Stream<G, (P, Vec<E>)>;
    fn intersect(&self, &mut Stream<G, (P, Vec<E>)>) -> Stream<G, (P, Vec<E>)>;
}

The records carry more information around with them; information that used to be on the stack now needs to be put in the records themselves. For example, we indicate the relation with the least count by a triple (prefix: P, count: u64, index: u64), data that would otherwise be in local variables. The signature of count is also changed to take and produces triples, like updating stack variables.

Although we are going to use this interface, you don’t need to know too much about about this. The main thing to know is that there are about fifty fairly predictable lines of code that go and implement a StreamPrefixExtender<G, P, E> for any type implementing PrefixExtender<P, E>.

impl<G, P, E, PE> StreamPrefixExtender<G, P, E> for Rc<RefCell<PE>>
where PE: PrefixExtender<P, E> {
    // the library does this for you, you just implement PrefixExtender.
}

Technically speaking, you will also need to tell timely dataflow how to distribute the prefixes. This will depend on how you distribute your relation, and is something I’ll say more about in an upcoming post.

Specific Join in Rust

With these abstractions, we are now ready to build a layer of the specific join algorithm. Before we do, let’s see what that would mean.

pub trait SpecificJoinExt<G, P, E> {
    fn extend(&mut self, extenders: Vec<Box<StreamPrefixExtender<G, P, E>>>)
        -> Stream<G, (P, Vec<E>)>;
}

We need to write a method for Stream<G, P> that, given a vector of arbitrary things implementing the StreamPrefixExtender<G, P, E> trait, produces a stream of pairs (P, Vec<E>). Also, we have to do it in the smart way described above, otherwise we’ll go slow like all the creaky database systems.

I’m just going to show you the code, but the comments should talk you through it. It’s just like we said.

impl<G, P, E> SpecificJoinExt<G, P, E> for Stream<G, P> {
    fn extend(&mut self, extenders: Vec<Box<StreamPrefixExtender<G, P, E>>>)
            -> Stream<G, (P, Vec<E>)> {

        // start with horrible proposals from a non-relation
        // ask each extender to try to improve each proposal
        let mut counts = self.select(|p| (p, 1 << 63, 1 << 63));
        for index in (0..extenders.len()) {
            counts = extenders[index].count(&mut counts, index as u64);
        }

        // for each of the extenders ...
        let mut results = Stream::empty();
        for index in (0..extenders.len()) {
            // find the prefixes the extender "won" the right to extend
            let mut nominations = counts.filter(move |p| p.2 == index as u64)
                                        .select(|(x, _, _)| x);

            // get the extensions and ask each other extender to validate
            let mut extensions = extenders[index].propose(&mut nominations);
            for other in (0..extenders.len()).filter(|&x| x != index) {
                extensions = extenders[other].intersect(&mut extensions);
            }

            // fold surviving extensions into the output
            results = results.concat(&mut extensions);
        }

        return results;
    }
}

This is the whole algorithm. It is really not super complicated. Rather, that is one layer of the algorithm. To fill out a full relational join we need to call extend multiple times, with different PrefixExtender objects wrapping the same relations, just for different lengths of prefix. Let’s do an example.

A low-latency triangle enumerator

If we define a graph as a set of pairs (src, dst), a triangle is defined as a triple (a,b,c) where (a,b), (b,c), and (a,c) are each in the set of pairs. We can think of the triangles query as a relational join over three relations, which are the same data just bound to different pairs of attributes.

Defining a PrefixExtender

We will represent a fragment of graph by a list of destinations and offsets into this list for each vertex. For each interval, we will keep the destinations sorted to make the intersection tests easier.

pub struct GraphFragment<E: Ord> {
    nodes: Vec<usize>,
    edges: Vec<E>,
}

We’ll just write a quick helper function to let use get at the edges associated with a node:

impl<E: Ord> GraphFragment<E> {
    fn edges(&self, node: usize) -> &[E] {
        if node + 1 < self.nodes.len() {
            &self.edges[self.nodes[node]..self.nodes[node+1]]
        }
        else { &[] }
    }
}

It is worth pointing out that Rust is doing some very clever things under the hood here. It notices that we are returning a reference to some memory, the type &[E], and that the only thing this could refer to is &self. Rust then sets up the lifetime bound for the output to be that of &self and will ensure that when we use the result it is not allowed to out-live self itself.

I’m going to lie a little and present a simplified sketch of the PrefixExtender for GraphFragment. The simplified version uses a reference-counted GraphFragment, all that Rc<RefCell<...>> stuff. This allows us to have just one copy of the graph loaded and to share it out between folks who need it. We also need a helper function of type L: Fn(&P)->u64 to extract a node identifier from the type P.

impl<P, E, L> PrefixExtender<P, E> for (Rc<RefCell<GraphFragment<E>>>, L)
where E: Ord, L: Fn(&P)->u64 {
    // counting is just looking up the edges
    fn count(&self, prefix: &P) -> u64 {
        let &(ref graph, ref logic) = self;
        let node = logic(&prev.0) as usize;
        graph.borrow().edges(node).len() as u64
    }

    // proposing is just reporting the slice back
    fn propose(&self, prefix: &P) -> Vec<E> {
        let &(ref graph, ref logic) = self;
        let node = logic(prefix) as usize;
        graph.borrow().edges(node).to_vec()
    }

    // intersection 'gallops' through a sorted list to find matches
    // what is "galloping", you ask? details coming in just a moment
    fn intersect(&self, prefix: &P, list: &mut Vec<E>) {
        let &(ref graph, ref logic) = self;
        let node = logic(prefix) as usize;
        let mut slice = graph.borrow().edges(node);
        list.retain(move |value| {
            slice = gallop(slice, value); // skips past elements < value
            slice.len() > 0 && &slice[0] == value
        });
    }
}

That was pretty easy, huh? It sure was a lot messier before I wrote that edges(node) helper method. It is also a bit grottier when I’m not lying about how things work, but let’s not let that get between us.

In the interest of completeness (and eyeballs on my code) let’s look at the implementation of gallop. From an input slice and value, it skips forward in exponentially increasing steps, and then once it would overshoot the target value it moves forward in exponentially shrinking steps.

// advances slice to the first element not less than value.
pub fn gallop<'a, T: Ord>(mut slice: &'a [T], value: &T) -> &'a [T] {
    if slice.len() > 0 && &slice[0] < value {
        let mut step = 1;
        while step < slice.len() && &slice[step] < value {
            slice = &slice[step..];
            step = step << 1;
        }

        step = step >> 1;
        while step > 0 {
            if step < slice.len() && &slice[step] < value {
                slice = &slice[step..];
            }
            step = step >> 1;
        }

        &slice[1..]   // this shouldn't explode... right?
    }
    else { slice }
}

The gallop method can use any slice as input, and we give it a slice cut out of the graph’s edge list. If we mis-use the result slice there could be a horrible risk of aliasing, data races, de-, or re-allocation. Rust can confirm that none of this happens and just lets us use the same memory. It does this through retain’s closure, through a method I wrote (gallop), and through lots of weird logic. This is great!

Building the triangles query

So to recap, we have an implementation of PrefixExtender<P,E> whenever we have a combination of GraphFragment<E> and L: Fn(&P)->u64. This second part, the function, is what tells us how to turn a prefix P into an integer we can use to look up extensions in the graph. We use the same graph data everywhere, but fake out multiple relations by letting the function from prefix to attribute vary.

That being said, let’s count some triangles! We’ll do this in parts. I’m going to lie a bit again, and present some simplifications. Nothing horrible, just possibly mysterious things that would require a diversion to rationalize (and may just be bad design on my part).

First we just write some code that from a Communicator, which indicates a worker’s index and its number of peers, figures out what fragment of the graph this worker will load up and be responsible for. The code then prepares a dataflow computation and an input into which we will feed a: u32 values. This is all timely dataflow boiler-plate, and not super exciting from an algorithmic point of view.

fn triangles<C, F>(communicator: C, graph_source: F)
where C: Communicator,
      F: Fn(u64,u64)->GraphFragment<u32> {

    // load up the slice of graph corresponding to our index out of peers.
    let graph = graph_source(communicator.index(), communicator.peers());
    let graph = Rc::new(RefCell::new(graph));

    // prepare a new computation with one input.
    let mut computation = new_computation(communicator);
    let (mut input, mut stream) = computation.new_input();

The next step is to create the extender from a to (a,b). We just need to pair a copy of graph with a function that converts from a into a graph node identifier, and it will propose the adjacent b values.

//  // define an extender that uses 'a' to suggest x: '(a,x)' extensions
    let ext_b = vec![Box::new((graph.clone(), |&a| { a as u64 }))];
    let mut pairs = stream.extend(ext_b).flatten();

That flatten() method simply converts a Stream<G, (P, Vec<E>)> to a Stream<G, (P, E)>. It is just 17 lines of code somewhere, so hooray for timely dataflow being easy to use (succint, at least).

The next step is to create the extenders from pairs (a,b) to triples ((a,b),c). There are going to be two extenders, as we want values c such that both (a,c) and (b,c) are present in the graph. Each extender needs to take an (a,b) and identify an edge source, and there only two choices.

//  // define extenders using 'a' and 'b' to suggest x: (a,x) and x:(b,x)
    let ext_c = vec![Box::new((graph.clone(), |&(a,_)| { a as u64 })),
                     Box::new((graph.clone(), |&(_,b)| { b as u64 }))]

    let mut triangles = pairs.extend(ext_c).flatten();

Let’s also take a peek at what we see, printing out the triangles (observe is another 16 lines of code).

//  // take a peek at what gets produced.
    triangles.observe(|&tri| println!("triangle: {:?}", tri));

Ok, we’ve got the data loaded up, the computation defined, and are ready to go. Let’s triangularate!

//  // finalize dataflow structure
    computation.0.borrow_mut().get_internal_summary();
    computation.0.borrow_mut().set_external_summary(Vec::new(), &mut Vec::new());

    // introduce u32s to find triangles rooted from them
    for node in (0..graph.borrow().nodes.len()) {
        input.send_messages(&((), node as u64), vec![node as u32]);
        input.advance(&((), node as u64), &((), node as u64 + 1));
        computation.0.borrow_mut().step();
    }

    // close input and finish any computation
    input.close_at(&((), graph.borrow().nodes.len() as u64));
    while computation.0.borrow_mut().step() { }
}

This runs through each possible source node and produces all triangles starting from that vertex.

Didn’t you say “low-latency”?

Good point. Although we went through all nodes in order, to enumerate all triangles, we didn’t have to. We could just as easily have written the main loop as:

for epoch in (0..) {
    let node = read_u32_from_console(); // not a real function!
    input.send_messages(&((), epoch), vec![node]);
    input.advance(&((), epoch), &((), epoch + 1));
    computation.0.borrow_mut().step();
}

This reads input in from the user, fires that off to the dataflow, which prints out observed triangles asap. To give a sense for how promptly, let’s throw a timer in the standard loop up above, and start things up single-threaded on a fairly standard LiveJournal dataset:

Caveat: the code is barely tested and may be wrong; do not buy/sell anything based on these data.

enumerated triangles from (0..1) in 67947ns
enumerated triangles from (1..2) in 42738ns
enumerated triangles from (2..3) in 26064ns
enumerated triangles from (3..4) in 48006ns
enumerated triangles from (4..5) in 19921ns
...

So the time between injecting a node identifier and getting all the triangles back out is some tens of microseconds. These numbers get a little better with batching, cutting out some of the overhead:

enumerated triangles from (0..10) in 231301ns
enumerated triangles from (10..20) in 276721ns
enumerated triangles from (20..30) in 149940ns
enumerated triangles from (30..40) in 137285ns
enumerated triangles from (40..50) in 155781ns
...

And even better with even more batching, as it starts to look more like 10-15us amortized per vertex.

enumerated triangles from (0..1000) in 10129064ns
enumerated triangles from (1000..2000) in 12018695ns
enumerated triangles from (2000..3000) in 11323832ns
enumerated triangles from (3000..4000) in 11224684ns
enumerated triangles from (4000..5000) in 14810182ns
...

This program is meant to stress several components and reveal what is slow and needs some work. The underlying system still has lots of room to improve (I think).The numbers should only get better.

Triangles are soooo lame.

Triangles are so 2014! Everyone does triangles! Do something cooler!

// define extenders to add a 'd' coordinate connected to each of a, b, c.
let ext_d = vec![Box::new((graph.clone(), |&((a,_),_)| { a as u64 })),
                 Box::new((graph.clone(), |&((_,b),_)| { b as u64 })),
                 Box::new((graph.clone(), |&((_,_),c)| { c as u64 }))];

let mut quads = triangles.extend(ext_d).flatten();

In case you missed it, that was us defining a dataflow computation to enumerate 4-cliques.

enumerated 4-cliques from (0..1) in 164474ns
enumerated 4-cliques from (1..2) in 230093ns
enumerated 4-cliques from (2..3) in 40667ns
enumerated 4-cliques from (3..4) in 154921ns
enumerated 4-cliques from (4..5) in 67344ns
...

Not impressed? How about:

enumerated 5-cliques from (0..1) in 115570ns
enumerated 5-cliques from (1..2) in 311967ns
enumerated 5-cliques from (2..3) in 110915ns
enumerated 5-cliques from (3..4) in 126838ns
enumerated 5-cliques from (4..5) in 104806ns
...

What about:

enumerated 6-cliques from (0..1) in 107292ns
enumerated 6-cliques from (1..2) in 342537ns
enumerated 6-cliques from (2..3) in 93778ns
enumerated 6-cliques from (3..4) in 134948ns
enumerated 6-cliques from (4..5) in 87148ns
...

I can keep going (seriously, I wrote a program to do it).

Arbitrary graph motif finding (small subgraphs you want to find in a large graph) is really easy to write. I’m sure there is lots of smarter research on how to do it will, but this is worst-case optimal, at least.

Wrap-up

There is some seriously cool work going on in join research. It isn’t nearly as stale as I thought it was. But, importantly, these new algorithms need better systems than your bog-standard batch processors.

How many of your favorite graph processors are able to start handing back 6-cliques in less than a millisecond, with a memory footprint that is roughly just the graph itself? Not the one you use? Why not? Being bad at things stopped being cool in high school…

On the positive side, in addition to this code, Naiad can do all this, and Flink should be able to as well.

From my point of view, I now have a cool problem I can use to tune timely dataflow. I would expect it to start getting better and more usable (and I’ll probably actually try it in more configurations now too).

Footnote: Skew

One of the theoretical concerns about this sort of approach is how well it distributes in the presence of skew. Each of our count, propose, and intersect operators is “data-parallel”, but is this actually helpful when the data are themselves not uniformly distributed among workers.

The implementation I’ve talked about is not skew-resilient. If a vertex has a really high degree, all the edges may be placed on one machine. Adding more machines will not make propose go any faster, and will not prevent all proposals from being sent to this one machine for intersection.

However, you can whip together a skew-resilient version of each of these operations.

count is naturally skew resilient because we just need a number for each prefix. They can be distributed (uniformly) by a hash on the prefix.
propose can be made skew resilient by distributing the extensions for each prefix to machines sequentially from a machine identified by a hash of the prefix. We need to know how many machines to ask for extensions, but each proposal should come with a count (I dropped it; oops).
intersect can be made skew resilient by distributing the (P, E) records by hash, and shipping proposals to the known location for each relation.

I’m not intimate enough with the work in the area to know for sure that this ticks all of the desired boxes. For example, one desideratum is that the computation take few “rounds”, in the MapReduce sense. Given that I don’t use MapReduce and still get computation done, this doesn’t really bother me.

Flip comments aside, it would be good to understand if this approach does represent a scalable, skew-resilient worst-case optimal join algorithm. I’ve been told that this would be pretty cool.

Credits and on-going work

This area (worst-case optimal join processing) is super cool, and I owe a debt to Semih Salihoglu and Chris Ré for introducing the two of us. These folks and their colleagues are still very active, pushing the boundaries of what can be done to make join processing faster and faster.

Semih, Chris and others have an approach to make even more complex joins efficient by using hypertree decompositions of queries. “Tree-like” acyclic queries have knowns ways to process them optimally (Viterbi-style). Once you get cycles they break, but if you can decompose the query into a tree of small cyclic queries and use the worst-case optimal techniques on the cyclic queries you get some nice results.

Chris Aberger, Chris and others have a neat way to make the intersection testing much more efficient using SIMD instructions, and show that they can get orders-of-magnitude speed-ups by using these techniques. Various “unnamed” proprietary vendors seem to get well and thoroughly trounced.

There is plenty of other work I’m not as familiar with (yet), but it seems to just keep coming. Be excited!

Differential dataflow

Tue, 07 Apr 2015 14:00:00 +0000

I’m now in Berlin, working on timely dataflow in Rust. It is up and running, multiple processes and all, so the next step is to get some experience using it to point out performance and usability issues. Unfortunately, at the moment there is a Rust internal compiler error preventing me from doing this, so I’m going to write a bit about some of the applications!

Differential dataflow is an approach to doing incremental data-parallel computation that several of us introduced at CIDR 2013. I’ll work through the details, but the rough idea is rather than have a point in the computation (a dataflow vertex) maintain a dataset, it maintains a collection of differences from which the data can be efficiently updated. If this sounds a bit like incremental dataflow computation, it is, and differential dataflow can be viewed as a generalization of existing techniques from incremental dataflow computation to dataflow graphs with possibly nested iteration.

Setting

Differential dataflow is primarily motivated by its ability to efficiently compute and update arbitrarily nested iterative data-parallel computations (things like MapReduce, but extended with an iteration operator). Each operator in a differential dataflow computation is defined much like a MapReduce stage, including:

For each input i a key function Key_i: Si -> K from source records to a common key type.
A reduction fuction Reduce: (K, &[S0], &[S1], ... , &[Sk]) -> &[R] from a key and some source records to some result records.

The output collection for any input collection is whatever results from grouping the source records by the key function, applying the reduction function, and merging the result records across all groups.

These operators are arranged in a graph, with outputs from one leading in to the inputs of others.

To this we add an iteration operator on collections, which takes an iteration function from collections to collections and a number of iterations. If we were to write this in Rust it might look like:

trait IterableExt {
    // returns something equivalent to iteration^steps(data)
    fn iterate<F: Fn(Self)->Self>(data: &Self, body: F, steps: u64) -> Self;
}

This iteration operator starts to make dataflow look like a pretty powerful computational framework, but implementing it efficiently raises all sorts of questions.

Naive implementation

Clearly there are some ways we could implement iterate for a Collection<S>. We could just have a for-loop which does as many steps as requested:

impl<S> Iterable for Collection<S> {
    fn iterate<F: Fn(Self)->Self>(data: &Self, body: F, steps: u64) -> Self {
        let mut data = data;
        for _ in (0..steps) {
            data = body(data);
        }
        return data;
    }
}

This approach is possibly very wasteful, especially in a not-uncommon case where the collection stabilizes as it iterates, reaching a fixed point (where each application of body results in the same collection). We could put some logic in place checking each iteration to see if it has change and exiting early if it hasn’t, but we may still be leaving a lot of performance on the table if most of the collection is not changing.

Incremental dataflow

If we only want to do one iterate at a time (“obviously”, you say; bear with me), a smarter approach is to frame the implementation in terms of incremental updates to the collections. Rather than fully form collections data each iteration, the operators could remember the previous values of data and we could instead determine and communicate only changes to data. If all operators are implemented this way, the entire computation can be in terms of these differences.

Incrementally updates are not only more efficient for communication, but in the data-parallel setting they can also dramatically improve performance. If I have a collection of one billion records, and I am presented with a single change: “add/remove record”, I only need to apply the key function to record and incrementally update the group that record might belong to. Other groups have not changed, and our computational model requires their outputs be functions only of the inputs. The only possible change to the output is due to changes to the key(record) group.

Limitations

One way to view incremental computation is that for a sequence of collections, it determines and communicates the differences between collections. There are several cases where this is a perfect fit, but also many where the requirement that the collections form a sequence is limiting.

Consider the problem of incrementally updating the input to an iterative computation. While the iterates of the iterative computation do form a sequence, we would now like to change the first element in that sequence. What incremental computation allows us to do is update the final element of the sequence. For some computations this may be correct, but for many computations it is simply incorrect to “patch” the output; it doesn’t produce the right answer.

Consider a hypothetical case where a bank uses some fraud data and a set of financial transactions to expand its view of “risky customers” to those people connected to a fraudster through any number of financial transactions. Perhaps you are accidentally determined to have had fraudulent activity, leading to your associates being labeled as fraudsters too. While you might correct the data after talking with the bank, removing your name from the source fraud dataset, your associates are still labeled as fraudsters and in one iteration you will be again too. In fact, none of you should be labeled as such, but updates to the current state cannot fix that. We must “restart” the iterative computation with the change.

Differential interpretation

Differential dataflow removes the restriction to sequences of collections by generalizing the notion of “differences between collections”. Formally, it uses a technique called “Moebius inversion” to allow differencing along an arbitrary partial order (rather than just a sequence). Although this sounds very complicated, and the area is deep and interesting, our application of it really isn’t so mysterious.

Imagine we have a set of collections Collection[t] indexed by some elements t of a set. If we avoid being too specific about this set, some examples become a bit easier to understand.

For our example of propagation of fraud data along edges in a financial transaction graph, the collection indicating who is or isn’t risky could be indexed by the pair (epoch, iteration), indicating the state of the collection using the epoch version of input fraud data, after iteration rounds of computation. The risk data are initially indexed by (0,0) and perhaps converge to their limit at (0,1032). You place a phone call to the bank, who can then update the base fraud data with index (1,0) indicating new data and no iteration yet. This data can then lead to a revised output now indexed by (1,937) (as perhaps the number of rounds to converge has decreased without you in the input data).

Pedagogically, the collection at (1,0) is more similar to the collection at (0,0) than it is to the collection “just a moment ago” at (0,1032). Rather than attempt to describe the collection at (1,0) in terms of the collection at (0,1032), as incremental computation might, we will describe a more flexible differencing scheme that instead relates it to the collection at (0,0).

More generally, we can describe the collection at (1,i) by the collections at (0,i) and (1,i-1).

The indices we choose to use, and the partial order we define over them, will drive how the differencing is done. By choosing an ordering that better reflects the development and dependence of data we are able to have fewer and smaller differences. The fact that (1,0) and (0,1032) are not ordered using the standard partial order allows us to avoid using either as part of the definition of the other.

Using the product partial order (one pair is less-or-equal another if both coordinates are), we get a nice alignment with the actual dependencies in an iterative computation, where an iterate depends only on collections at prior iterations and prior epochs.

Differential representation

Let’s write down a fairly uncontroversial view of how we might represent collections Collection[t] as the accumulation of differences Difference[t] over a common ordered set of indices:

Collection[t] = sum_{s <= t} Difference[s]

This feels very safe and intuitive for a total order like the natural numbers: we add up the differences up to and including index t.

Nothing about this equation requires that the elements be totally ordered. We can just as easily use a partial order like (u64, u64): pairs of numbers where one is less or equal another exactly when both coordinates are, just like in our (epoch, iteration) example above.

Differential computation

Our intuition for computation may start to break down without a total order, raising the technical question of “how do we determine the values of Difference[t]?” A bit of mathematical re-arrangement takes the implicit definition of Difference[t] above to

Difference[t] = Collection[t] - sum_{s < t} Difference[s]

which indicates that we can derive differences from the collection at index t and all “earlier” differences. This adds a technical requirement that the partial order be “well-founded”, which informally means that we can always find something to start the differencing from. We will make this a lot easier by assuming that our computation is driven by a system (a timely dataflow system, perhaps!) presenting differences in an order so that for each t we have already determined and recorded Difference[s] for all s < t.

This may look very expensive, and of course if we had to compute Difference[t] for every t we would be pretty upset. Fortunately, as with incremental computation, when an operator is presented with new input differences, we can restrict the t for which it may possibly produce output differences. Further, our use of partial orders can yield substantially sparser differences, meaning less work to do.

Like in incremental dataflow, an differential dataflow operator only needs to update the output for keys associated with changed inputs. However, an update at an index t may require outputs at indices other than t, which is very much un-like incremental computation. For the moment, let’s just say that we can compute these indices efficiently, and the number is often not particularly large.

Implementation

Let’s go through a simple implementation of a differentail dataflow operator (in Rust).

Representing collections

The first thing to define is a differential representation of a collection. In our simple approach we will just store each difference[t] as a list of (V, i64) indicating the change to the count associated with each record. In fact, we’ll just stash all of these in the same Vec<(V, i64)>, and use a second array to delineate which regions are associated with which indices, of index type T.

struct CollectionAsList<T, V> {
    updates: Vec<(V, i64)>,   // value and delta
    indices: Vec<(T, usize)>, // index and count
}

What do we need to be able to do with this representation? So far, I’ve been using traits for accessing the data as indexed collections or as indexed differences:

trait Collection<T, V> {
    fn setc(&mut self, index: T, data: &mut Vec<(V, i64)>);
    fn getc(&self, index: &T, data: &mut Vec<(V, i64)>);
}

Here the setc method takes an index and an intended collection, and inserts the correct differences at index to ensure that when accumulated the result is data. The getc method populates the supplied Vec<(V, i64)> with the accumulated differences, which by definition is the value of the collection at index.

trait Difference<T, V> {
    fn setd(&mut self, index: T, data: &mut Vec<(V, i64)>);
    fn getd(&self, index: &T) -> &[(V, i64)];
}

In the Difference trait, the setd method just internalizes the suppied vector of updates and the getd method just returns the slice corresponding to the requested index.

The method names have c and d suffixes so that I don’t have to show you Rust’s name resolution. It isn’t that horrible, but it is distracting I’ll probably just end up changing the traits anyhow.

In my implementation I’m currently using the constraint that V: Ord, meaning we can tell when two instances are the same (important for accumulating differences) and there is some ordering on them. This ordering property is less obviously necessary, but it does allow me to do merge-based accumulation, rather than faking out a hash table inside the arrays or something horrible like that.

Representing computations

The next thing to consider is how an operator might be implemented. In the interest of manageable type signatures we will do a unary operator, from collections of S to collection of R, indexed by T.

For each key, we will need to keep track of the accepted source differences, and we will want to keep track of the produced result differences. We will use the following simple structure for per-key differential state:

pub struct DifferentialShard<S, R, T> {
    source: CollectionAsList<T, S>,
    result: CollectionAsList<T, R>,
}

Next we’ll want to put these together as an operator, involving a key selection function, some logic to apply to transform source records to result records, and some storage to make it all stick together.

pub struct DifferentialOperator<K: Hash+Eq, S, R, T, L, KF> {
    key_fn: KF,
    logic:  L,
    shards: HashMap<K, DifferentialShard<S, R, T>>,
}

This isn’t particularly instructive yet. Let’s constrain the types and implement some methods.

impl<K, S, R, T, L, KF> DifferentialOperator<K, S, R, T, L, KF>
where K: Hash+Eq,
      S: Ord+Clone,
      R: Ord+Clone,
      T: PartialOrd+Clone,
      L: Fn(&K, &[(S,i64)], &mut Vec<(R,i64)>),
      KF: Fn(&S)->K {
  // .. rest of implementation broken out below ..

These constraints mean that our data S and R need to be totally ordered and clonable, and our index T needs to be partially ordered and clonable. We also have types KF and L corresponding to closures for our key function and operator logic. The key function must take a reference to a source record and produce a key, the operator logic must take a reference to a key, a reference to source data, and a populatable vector for result data. The logic is asked to work on pairs (S,i64) and (R,i64) in order to make my life easier, rather than S and R followed by accumulating the counts.

Next we write a method to accept new input data, not yet grouped by key, which does the partitioning of source records and loads each of the groups into the appropriate DifferentialShard.

fn accept_source(&mut self, index: T, data: Vec<(S, i64)>) {
    let mut stage = HashMap::new();
    for (record, delta) in data {
        let key = (self.key_fn)(&record);
        let mut list = match stage.entry(key) {
            Occupied(x) => x.into_mut(),
            Vacant(x)   => x.insert(Vec::new()),
        };
        list.push((record, delta));
    }
    for (key, mut list) in stage.drain() {
        let shard = match self.shards.entry(key) {
            Occupied(x) => x.into_mut(),
            Vacant(x)   => x.insert(DifferentialShard::new()),
        };
        list.sort(); // using orderedness to merge
        shard.source.setd(index.clone(), &mut list);
    }
}

In principle, we could be stashing the input records at the ends of shard.source.updates rather than staging them first, but we would need to tweak the interface a bit to expose more than just setd.

Finally, for now, we write a method to relate the source collection to result collection at a specific index, and to correct the result differences if they do not currently accumulate to the correct result collection.

fn update_result(&mut self, index: &T, output: &mut Vec<(R, i64)>) {
    let mut temp_src = Vec::new();
    let mut temp_dst = Vec::new();
    let logic = &self.logic;
    for (key, shard) in &mut self.shards {
        temp_src.clear(); temp_dst.clear();

        // fetch collection, run logic, compare
        shard.source.getc(index, &mut temp_src);
        if temp_src.len() > 0 {
            logic(key, &mut temp_src, &mut temp_dst);
        }
        shard.result.setc(index.clone(), &mut temp_dst);

        for result in shard.result.getd(time)  {
            output.push(result.clone());
        }
    }
}

The main possibly surprising part (the part I got wrong the first time) is that we only want to call the logic on non-empty temp_src collections. This is an important part of correctness, as everything goes horribly wrong if we let logic produce output on empty collections; we don’t want to have to run it for all possible keys, but rather only the ones for which we’ve actually seen data.

Trying it out!

Oh yeah! We wrote all that code, let’s see what it does. Here is an operator which counts the number of distinct strings with each length, producing a weird (String, usize) result pair for each length.

fn main() {
    let mut vertex = DifferentialOperator::new(|x:&&str| x.len(), |k, s, t| {
        t.push(((format!("length: {:?}", k), s.len()), 1))
    });

    let mut result = Vec::new();
    let sched = vec![(Product::new(0,0), vec![("a", 1), ("b", 3), ("cc", 2)]),
                     (Product::new(0,1), vec![("a",-1), ("b",-3)]),
                     (Product::new(1,0), vec![("a",-1), ("b",-1)]),
                     (Product::new(1,1), vec![("a", 1), ("b", 2)])];

    for (index, differences) in sched {
        result.clear();
        vertex.accept_source(index.clone(), differences);
        vertex.update_result(&index, &mut result);
        println!("difference[{:?}] : {:?}", &index, result);
    }
}

Let’s go through the produced output line by line:

First we see here the changes from the initially empty result collection to the output on the supplied source collection. We have two length-one strings and one length-two string:

difference[(0, 0)] : [(("length: 1", 2), 1), (("length: 2", 1), 1)]

Next we see what happens when we remove the length-one strings: we lose any record of “length: 1”:

difference[(0, 1)] : [(("length: 1", 2), -1)]

The next change is in a different coordinate, and doesn’t include the removal of the length-one strings that occurs at (0, 1). Rather, it removes one of the two length-one strings and decrements the number of the other by one (though two occurences of “c” should still remain). The result is one distinct length-one string, and the way differential dataflow represents that is to subtract one record and add the other.

difference[(1, 0)] : [(("length: 1", 1), 1), (("length: 1", 2), -1)]

Although we might like to change the 2 to a 1 by directly subtracting one, those two values are now part of the data rather than the counts. If they had been formatted into the string instead, it would be clearer that this is not possible.

Finally, we add back an “a” (as otherwise the total would be negative) and leave ourselves with one “b”. Index (1,1) has one length-one string, and so the output is naturally:

difference[(1, 1)] : [(("length: 1", 2), 1)]

Yeah, that last one isn’t obvious at all, is it?

We would like the output to be [(("length: 1", 1), 1), (("length: 2", 1), 1)], and the difference above is the right quantity to add in to make this be the case. We subtracted too many [(("length: 1", 2), 1)] records, at both (0, 1) and (1, 0), and need to put one back.

Things yet to do

The main issue that this example avoids is knowing on which indices to call vertex.update_result.

For example, even if we had not had any input differences at (1, 1), which would be odd as the counts for both “a” and “b” would go negative, we would still need to call update_result. The input difference at (1, 0) introduces the need to at least check the output at (1, 1).

If we just call update_result at (1, 1), without adding source differences, we get the following:

difference[(1, 1)] : [(("length: 1", 1), -1), (("length: 1", 2), 2)]

Here we are told that we have too many reports of one length-one string, and need multiple reports of two length-one strings. This is because we were already short one such report, and there are now two length-one strings (though with negative counts; not something we paid attention to in our logic).

Roughly, difference[t] can lead to result differences at t joined (in the partial order sense of least upper bound) with the join of any subset of pre-existing indices with non-empty differences.

We should also be more careful about noting the subset of keys that need updating at a given index (we currently update them all. Naughty!). This isn’t much more complicated than maintaining a HashMap<T, HashSet<K>>, populated by the join logic described above.

None of this is too complicated to compute, but stitching it all together properly requires something like a timely dataflow system, where one can respond to source differences with a combination of result differences and notification requests (for checking possible result updates at future indices).

Ideally the Rust ICE blocking me wiring all this together gets sorted out promptly, otherwise I will be forced to delight you with a post about worst-case optimal join processing in timely dataflow.

Bigger data; same laptop

Wed, 04 Feb 2015 14:00:00 +0000

This post follows up the previous post, “Scalability! But at what COST?”, which got a great response. The short version of the previous post is that for the graph datasets and computations the scalable systems research community is currently looking at, a laptop outperforms the scalable systems.

There were several flavors of response to the post (many of which were very supportive!), but one that I’d like to address in this post is

One billion edges is just a few gigabytes; it’s hardly “big data” …

My personal opinion is that if your system doesn’t perform well on one billion things, you don’t yet have the credibility to claim it’s going to work much better on a trillion things. But let’s see where this goes…

The only reported data I know of for graph processing at the “one trillion things” scale is Facebook’s processing of a one trillion edge graph, where they got the per-iteration pagerank time down to under four minutes on 200 commodity machines (number of cores unknown to me). That’s really quite good (and, much better than Giraph performed in GraphX’s evaluation).

I don’t have a trillion edge graph, unfortunately, but thanks to the nice people at Web Data Commons, I was able to download a 128 billion edge graph, based on the hyperlink structure from Common Crawl. Using the same algorithms from the previous post yields a new (mostly empty) column:

Twenty pagerank iterations
System	cores	twitter_rv	uk_2007_05	common crawl
Spark	128	857s	1759s	unknown
Giraph	128	596s	1235s	unknown
GraphLab	128	249s	833s	unknown
GraphX	128	419s	462s	unknown
Single thread	1	242s	256s	46600s*

*: An earlier version had 23,653 seconds as the number for pagerank. Unfortunately, the code was set to do only ten iterations (in order to compare to GraphX, whose scaling measurements are for only ten iterations). We apologize for the error.

Graph connectivity
System	cores	twitter_rv	uk_2007_05	common crawl
Spark	128	1784s	8000s+	unknown
Giraph	128	200s	8000s+	unknown
GraphLab	128	242s	714s	unknown
GraphX	128	251s	800s	unknown
Single thread	1	15s	30s	1700s

Importantly, these algorithms are exactly the same as in the previous post. Nothing has been changed, other than (a few sections down) a different on-disk representation of the graph so that it fit on my SSD. The edges per second are mostly within a factor of two across the graphs; this approach even scales!

Note: The single-thread numbers for pagerank on the smaller graphs are not the same as in the previous post. Because the common crawl dataset is so large, I was only able to process it in Hilbert curve order. It seemed appropriate to use the corresponding numbers for the smaller graphs, rather than the vertex order numbers.

A COST challenge for scalable systems

I’d like to challenge the teams behind the existing graph processing platforms to evaluate and report their COST (Configuration that Outperforms a Single Thread) for this dataset. The dataset is nearly two orders of magnitude larger than those most currently use for system evaluation, and yet still nearly one order of magnitude smaller than can be easily processed by my laptop. It is much “bigger” data than we’ve been using previously, and it is available for everyone to download, so get to it!

The only partial data we have is for Giraph, for which 200 machines is more than enough. I don’t think we know very much about any other scalable graph processing system. Let’s fix that!

I’ll happily post updates for performance numbers on specific systems. To make it fun, let’s say that each submission gets to name another system whose row is “unknown” until its measurements arrive.

Implementation notes

There were some relatively minor implementation details involved in getting 128 billion edges onto my laptop, and then processed efficiently (which isn’t to say that I got them right the first time). Doing the math, if one writes down a (u32, u32) for each edge, this ends up at a terabyte of data, and my laptop doesn’t have that kind of SSD. So, we’ll have to be a bit smarter.

Using the Hilbert curve layout leads to a very simple compressed representation: having sorted the u64 values it uses to represent edges, one can simply delta-encode them (writing the differences between adjacent values). Because the Hilbert curve teases out locality, the gaps are often quite small, and therefore compress nicely. When I combined a variable-length integer encoding with gzip, the resulting file was only 45GB; about 2.8 bits per edge). Decompressed (with just the variable-length delta encoding), it is 154GB, which is the representation I used to get the measurements above.

Not only is the Hilbert curve good for compression, it is pretty much mandatory due to the locality it provides when accessing the per-vertex state. Random access into main memory is not all that bad, but when the working set for vertex state in pagerank is 40GB, random access turns in to thrashing. This is not only slow, but it wears out my SSD, and no one wants that. The near-linear performance scaling as we went up two orders of magnitude in size is evidence for how cool the Hilbert curve is.

Transforming the data to the compressed Hilbert curve encoding was relatively efficient. My laptop was able to transform (encode, sort, and compress) the data faster than I could download it, so I didn’t work too hard to optimize the performance. However, this is an excellent example of where parallelism can help: each of the 700 files comprising the graph could be processed independently (and I did fire up a second process on occasion). If you had 100 cores (like most of the scalable systems use) the whole conversion process would just take a few minutes.

The implementation is available in the COST repository, though the documentation is not yet brilliant. If you have questions, especially if you’d like to be doing this sort of thing in your scalable system and want pointers, just shoot me an email.

What’s next

I’m not going to try this nonsense on any larger graphs. The ball is now in the court of the scalable systems. We’ll just have to wait and see how they do before setting any stretch performance goals.

Instead, it’s time to get back to work on timely dataflow in Rust! Stay tuned for that kind of post.

Scalability! But at what COST?

Thu, 15 Jan 2015 14:00:00 +0000

Michael Isard, Derek Murray, and I recently sent in a HotOS submission (it’s not blind, so no harm talking about it, we think). The subject is hinted at from the post title (stolen from the paper title):

Big data systems may scale well, but this can often be just because they introduce a lot of overhead.

Rather than making your computation go faster, the systems introduce substantial overheads which can require large compute clusters just to bring under control.

In many cases, you’d be better off running the same computation on your laptop.

Methodology

Here is the set-up: we took several recent graph-processing publications from the systems community, and compared the measurements they report to simple single-threaded implementations running on my work laptop (RIP). We wrote competent implementations, but we didn’t obsess deeply over fancy algorithms, cunning data-dependent tweaks, or what have you. I’ll show you the code, and you decide.

We evaluated PageRank (20 iterations) and graph connectivity on two graphs, twitter_rv and uk_2007_05. We chose these algorithms and datasets mostly because a recent OSDI 2014 paper GraphX used exactly these to evaluate several of the top systems, and we wanted to borrow their numbers rather than try to reproduce them on systems we aren’t expert with.

As a caveat, these algorithms are quite specific to graph processing, and the data sets are not large (billions of edges, but still just a few gigabytes). Our conclusion is not that the systems we are going to look at are obviously bad, but rather that there is not yet much evidence that they are especially good.

The point we try to make in the HotOS submission is that evaluation of these systems, especially in the academic context, is lacking. Folks have gotten all wound-up about scalability, despite the fact that scalability is just a means to an end (performance, capacity). When we actually look at performance, the benefits the scalable systems bring start to look much more sketchy. We’d like that to change.

Measurements

All measurements other than the “single thread” lines are from the GraphX paper linked above. I’ll talk about the single-threaded implementations soon, but the results for the most direct implementations of PageRank and label propagation (the algorithm the systems use for graph connectivity) look like this:

Twenty pagerank iterations
System	cores	twitter_rv	uk_2007_05
Spark	128	857s	1759s
Giraph	128	596s	1235s
GraphLab	128	249s	833s
GraphX	128	419s	462s
Single thread	1	300s	651s

Label propagation to fixed-point (graph connectivity)
System	cores	twitter_rv	uk_2007_05
Spark	128	1784s	8000s+
Giraph	128	200s	8000s+
GraphLab	128	242s	714s
GraphX	128	251s	800s
Single thread	1	153s	417s

These single-threaded numbers paint the scalable systems in a pretty dismal light. With 128x as many cores, none of the scalable systems consistently out-perform a single thread at PageRank, which is probably one of the simplest graph computations (sparse matrix-vector multiplications). The systems are almost a factor of two slower than the single-threaded implementation for label propagation.

Unfortunately, it is only going to get worse for the scalable systems as our investigation continues.

Implementation

Let’s talk a bit about what the implementations look like, so that we can figure out whether we are being totally unfair. First, the measurements were taken on a laptop I no longer possess. I’ve reproduced most of the code in Rust on my new laptop, and the results are mostly the same (some faster, some slower). The measurements above are from a C# implementation, so this isn’t a “Rust is super fasterest” issue. The code is available on GitHub, including both the C# version and the Rust version.

The algorithms we are talking about are all based on iterative scans of the edges in a graph. To support these algorithms, we are going to produce a few implementations of the following trait:

trait EdgeMapper {
    fn map_edges<F: FnMut(u32, u32)->()>(&self, action: F) -> ();
}

The map_edges method is generic with respect to a closure. It’s a mouthful, but it means action can be called multiple times on inputs of type (u32, u32), edges, and may modify its environment. This will probably make more sense once we see it used.

One nice thing about Rust is that each closure corresponds to a new type F, and a new specialized instance of map_edges, inlining the action method. In C# I had to manually unroll some loops. Ew.

Algorithms

Let’s look at how PageRank gets implemented with respect to such a trait:

fn pagerank<G: EdgeMapper>(graph: &G, nodes: u32, alpha: f32) {
    let mut src: Vec<f32> = (0..nodes).map(|_| 0f32).collect();
    let mut dst: Vec<f32> = (0..nodes).map(|_| 0f32).collect();
    let mut deg: Vec<f32> = (0..nodes).map(|_| 0f32).collect();
    graph.map_edges(|x, _| { deg[x] += 1f32 });
    for _iteration in (0 .. 20) {
        for node in (0 .. nodes) {
            src[node] = alpha * dst[node] / deg[node];
            dst[node] = 1f32 - alpha;
        }
        graph.map_edges(|x, y| { dst[y] += src[x]; });
    }
}

I’m not going to say very much about whether this is a good pagerank implementation. There are better versions, but everyone uses this algorithm for benchmarking to ensure they are measuring their system’s performance, not the algorithm’s. This is textbook PageRank, no bells, no whistles.

Label propagation is basically just as simple. For those of you unfamiliar with the algorithm, each node maintains a label, and repeatedly improves it by asking their neighbors if they have a smaller label. This is run until convergence, at which point all nodes in the same component have the same label.

fn label_propagation<G: EdgeMapper>(graph: &G, nodes: u32) {
    let mut label: Vec<u32> = (0..nodes).collect();
    let mut done = false;
    while !done {
        done = true;
        graph.map_edges(|x, y| {
            if label[x] != label[y] {
                done = false;
                label[x] = min(label[x], label[y]);
                label[y] = min(label[x], label[y]);
            }
        });
    }
}

These are pretty simple implementations. No tricks. Let’s look at how we implement an EdgeMapper.

EdgeMapper implementation

Our first EdgeMapper implementation will be based on an adjacency list representation, which we will just map into memory and iterate over. There will be a smarter one later on.

We’ll assume we have two files, graph.nodes and graph.edges, where

graph.nodes contains a sequence of (u32, u32) pairs representing (node_id, degree).
graph.edges contains a sequence of u32 values representing edge endpoints.

To enumerate the edges, we iterate through graph.nodes, where for each (node_id, degree) pair we read degree values from graph.edges and then pair each with node_id to form an edge.

Ignoring the TypedMemoryMap for the moment (it’s 32 lines of wrapper around Rust’s MemoryMap), the mapper structure and its implementation of EdgeMapper are:

pub struct NodesEdgesMapper {
    nodes:  TypedMemoryMap<(u32, u32)>,
    edges:  TypedMemoryMap<u32>,
}

impl EdgeMapper for NodesEdgesMapper {
    fn map_edges<F: FnMut(u32, u32)->()>(&self, mut action: F) -> () {
        let mut offset = 0;
        for &(node, count) in self.nodes[].iter() {
            let limit = offset + count as usize;
            for &edge in self.edges[offset..limit].iter() {
                action(node, edge);
            }
            offset = limit;
        }
    }
}

That’s all! (except for the TypedMemoryMap type, available in the repository).

Getting smarter

Perhaps the single-threaded implementations above are a bit too simple. There are some very easy modifications that result in significantly improved performance. Remember how the simple single-threaded implementations were on-par with or maybe slightly better than the scalable systems? Here are the improved results using smarter algorithms (still from my old laptop and C#):

Twenty pagerank iterations
System	cores	twitter_rv	uk_2007_05
Single thread (simple)	1	300s	651s
Single thread (smarter)	1	110s	256s

Label propagation to fixed-point (graph connectivity)
System	cores	twitter_rv	uk_2007_05
Single thread (simple)	1	153s	417s
Single thread (smarter)	1	15s	30s

There are two types of improvement, processing edges in different orders and using better algorithms, that put the single-threaded implementations so far ahead of the scalable systems.

Different data layout

The edge order used above processes all edges from one vertex before moving to the next vertex. Notice that neither of the algorithms need this to hold to be correct. A different ordering, one which exhibits much better cache locality, is the Hilbert curve ordering. Informally, it is like ordering edges (x,y) by the interleaving of the bits of x and y: you trade the great locality in the x coordinates to upgrade the poor locality in the y coordinate. You get (good, good) locality instead of (great, poor).

Visually, we can think of ordering pairs (x,y) as drawing a continuous line (a space-filling curve) through a 2d square. An initially square region is decomposed into four squares, each of which will be filled completely before moving to the next square. Notice that the sequence 0, 1, 2, 3 moves only to adjacent squares. The process is applied recursively within each square to order elements within, and even at the finer granularity the movements are only to adjacent squares, even when moving between squares at the coarser granularity. The ordering (curve) we produce is continuous, and very localized.

The Hilbert implementation of EdgeMapper is similar to our prior implementation, with some minor layout changes. We start from two files:

graph.upper contains a sequence of ((u16, u16), u32) values, each a 65536x65536 square indicating the upper 16 bits of x and y, followed by the number of edges in the square. These entries are ordered by the order their corresponding squares are visited by the curve.
graph.lower contains a sequence of (u16, u16) values for the lower 16 bits for each edge. These entries are ordered by the the order the curve visits them when traversing their square.

The EdgeMapper implementation is basically identical to the implementation for vertex-order above, but results in improved PageRank running times. One lesson is that PageRank is essentially memory-bound; the time is spent either thrashing the TLB or waiting for results from the memory controller, rather than the CPU or disk. It scales because each machine offers an indepedent memory controller. Instead, you can partially fix the problem by re-arranging memory accesses like we have done here. You could also use huge pages (the C# code does), which I can’t do yet on OS X.

We should also admit that the Hilbert curve ordering is not typically how graphs are presented, and consequently there would be some work involved in transforming your input data to this ordering. If you are only planning on doing one computation, it may not be worth the effort. Fortunately, a single-threaded implementation takes less time to re-order the edges (about 180s in Rust) than any of the systems above take to perform 20 iterations of PageRank. It’s included in the repository.

Both GraphLab and GraphX do their own data layout as part of pre-processing. One of the layouts they consider resembles the Hilbert ordering: partitioning vertices into k parts and sending sub-squares of edges to k^2 workers. The Hilbert-curve ordering is totally compatible with their system designs, and they could probably speed up a bit if they used it. It is somewhat in conflict with their approaches to sparse updates (when you only want to process a subset of vertices), trading latency in the sparse-update case for throughput in the (often more expensive) bulk-processing case.

Different algorithms

The label propagation algorithm is not commonly taught as a good algorithm for graph connectivity. It has poor asymptotic bounds, even when compared with more traditional algorithms from 90 years ago.

Consider the standard union-find algorithm, which maintains a root variable for each vertex, and scans the edges once, merging roots when it finds edges whose endpoints have different roots. We use the “weighted-union” variation, where the merge pays attention to how deep each root is when determining which way to swing the pointers (this gives it an O(m log n) asymptotic complexity).

fn union_find<G: EdgeMapper>(graph: &G, nodes: u32) {
    let mut root: Vec<u32> = (0..nodes).collect();
    let mut rank: Vec<u8> = (0..nodes).map(|_| 0u8).collect();
    graph.map_edges(|mut x, mut y| {
        while x != root[x] { x = root[x]; }
        while y != root[y] { y = root[y]; }
        if x != y {
            match rank[x].cmp(&rank[y]) {
                Less    => root[x] = y,
                Greater => root[y] = x,
                Equal   => { root[y] = x; rank[x] += 1 },
            }
        }
    });
}

This is one more line of code than label propagation, but it is 10x faster and 100x less embarassing.

The union-find algorithm is fundamentally incompatible with the graph computation approaches Giraph, GraphLab, and GraphX put forward (the so-called “think like a vertex” model). That doesn’t stop it from being way better than label propagation, and even parallelizable.

Important things

Several factors affect the performance we’ve seen above. Let’s see where the differences come from.

To get consistent measurements despite using mmap, each of these numbers are from a second run. This means that the data are effectively “in memory”. The first runs are sketchier and more variable. We’ll use graph connectivity as our example, from label propagation to our fastest union-find, running on my laptop (a 2014 MacBook pro) using Rust implementations.

150.6s	:	Label propagation and vertex ordering
31.1s	:	Switching to Union Find
16.0s	:	Switching to Hilbert-curve ordering
11.8s	:	Using unsafe indexing for roots
8.7s	:	Alternate EdgeMapper with smaller memory footprint (in the repo)
8.0s	:	Pre-fetching root[x] and root[y] before the while loop.

Note, these numbers differ from those reported above due to being a totally different implementation. Mainly, the first measurements took advantage of large pages, which do not seem to be exist on OS X. This benefit is largely recouped by using the Hilbert ordering, which also eases pressure on the TLB. The custom EdgeMapper re-uses a small buffer, and likely messes up the TLB less than mmap does.

Take-aways

Lots of people struggle with the complexities of getting big data systems up and running, when they possibly shouldn’t be using the systems in the first place. The data sets above are certainly not small (billions of edges), but still run just fine on a laptop. Much faster than the distributed systems, at least.

Here are two helpful guidelines (for largely disjoint populations):

If you are going to use a big data system for yourself, see if it is faster than your laptop.
If you are going to build a big data system for others, see that it is faster than my laptop.

We implemented a few of these algorithms in Naiad and were able to get performance that beats the single thread. Admittedly, it took some work for parallel union-find, but we learned quite a bit and made something better as a result (delightfully: we didn’t have to change Naiad itself; just application logic).

We think everyone should have to do this, because it leads to better systems and better research.

Of course, there are some reasons why you might expect some overheads for your distributed system. The machines Amazon provides as VMs are differently spec’ed than my laptop, which targets peak single-threaded performance. GraphX uses hash tables (which can add an order of magnitude to the critical path) rather than dense arrays, to maintain its per-vertex state, to provide a layer of robustness. Fault-tolerance calls for data to be pushed from fast random access memory out to stable storage to insure against lost work in the case of a failure.

At the same time, it is worth understanding which of these features are boons, and which are the tail wagging the dog. We go to EC2 because it is too expensive to meet the hardware requirements of these systems locally and fault-tolerance is only important because we have involved so many machines, at which point we wonder whether the robustness of these hash tables is worth all the additional effort. Perhaps, but we should be thinking about it quite a bit harder than we currently are.

Timely dataflow: core concepts

Mon, 29 Dec 2014 16:00:00 +0000

This is the second post in a series on timely dataflow. The first post overviewed some of the context for timely dataflow, and suggestions at new directions. This post will start to detail some of the moving parts of the new hierarchical approach.

Our plan is to organize timely dataflow graphs hierarchically, into nested scopes, each of which can be implemented by other timely dataflow graphs. This has several advantages, mostly due to the layer of abstraction hiding details that other parts of the outer timely dataflow graph need not know. On the other hand, we will need to think carefully about the abstraction boundary to avoid giving up precision.

Approach

Whereas Naiad maintained a flat view of the timely dataflow graph, with counts for each (location, timestamp) pair indicating unfinished work, we will need to take a different approach. Rather than present such fine detail about where unfinished work lives to all participants, we will use the hierarchical structure to simplify the information.

Each scope will project its unfinished work to its outputs, reporting for each output the timestamps at which messages might possibly emerge. As it performs work, it will communicate progress to the outer timely dataflow graph in the form of changes to the counts for (output, timestamp) pairs.

Similarly, unfinished work and progress made elsewhere in the graph is first projected to the scope’s inputs before it is communicated to the subgraph.

An example

To take an example we borrow an enormous figure from the Naiad paper.

In this example we have two contexts, or “scopes” as we’ll call them: the outer “streaming” scope and the inner “loop” scope. While the inner loop scope may progress rapidly, in the outer streaming scope vertex D only needs to be informed when some externally visible progress is made. Specifically, when the output of the inner loop scope will no longer produce messages bearing a specific timestamp.

In fact, the outer streaming scope doesn’t need to know much of anything about the inner loop scope, other than it has one input and one output, and whenever it makes externally visible progress.

Our plan is just to structure the progress tracking mechanisms along these lines, reporting progress upwards having projected out any irrelevant detail and coalesced (and often cancelled) the results.

Generally, there are other things it will be helpful for the outer scope to know that this figure doesn’t demonstrate especially well. If the inner scope had multiple inputs and outputs, it would be helpful to know which inputs connected which outputs. If the inner scope had an exogenous source of input data, it would need to warn the outer scope to expect it. We will sort out these details.

Core traits

Three traits lie at the core of our timely dataflow approach: Timestamp, PathSummary, and Scope. Each scope has its own timestamp and path summary types, implementing the corresponding traits.

The Timestamp trait

The Timestamp trait represents a type suitable for use as a message timestamp. In Naiad this role was played by a sequence of unsigned integers. We plan to admit more general timestamps, but with some constraints, specifically (from the code):

trait Timestamp: PartialOrd+Eq+PartialEq+Copy+Default+Hash+Show+Send+'static { }

This is something of a mess, and strictly speaking not all of the constraints are necessary. Just helpful.

The most relevant constraint is PartialOrd, which provides for methods le, lt, ge, gt, for comparing two elements of the same type. Types implementing Timestamp must be partially ordered.

The PathSummary trait

The PathSummary trait is parameterized by a type implementing Timestamp. It indicates how the timestamp should be expected to advance as it moves from one location to another in a timely dataflow graph. It also provides support for combining two summaries (summarizing the concatenated paths).

trait PathSummary<T> : PartialOrd+Eq+Copy+Clone+Show+Default+'static
{
    fn results_in(&self, timestamp: &T) -> T;       // advances a timestamp
    fn followed_by(&self, other: &Self) -> Self;    // composes two summaries
}

Path summaries allow us to translate events at some distant location to the event’s implications locally. If a distant vertex finishes processing the last of its inputs and produces no outputs, which timestamp (or timestamps) should be decremented locally?

To summarize all paths from one location to another, we exploit the fact that types implementing PathSummary are partially ordered, and that we are interested in the earliest timestamps that might emerge. Any set of path summaries can be reduced down to those elements that are not strictly less than some other set element, which we facilitate with the Antichain<S> structure.

It is likely that the trait will be updated as additional constraints are discovered. For example, it is very likely that results_in should return either Option<T> or Iterator<T> types rather than single elements. Otherwise, it may be difficult to accommodate bounded loops that discard messages once their timestamp exceeds the loop bound. Other similar unsupported patterns likely exist.

The Scope trait

The Scope trait is where the pieces start to come together. A scope represents an element in a timely dataflow graph, as seen from the outside world (or its parent scope). A scope is parameterized by a timestamp type T and a path summary type S. To implement the trait, a type must provide several methods, which we will soon detail (some diagnostic methods are elided).

trait Scope<T: Timestamp, S: PathSummary<T>> : 'static
{
    fn inputs(&self) -> uint;   // number of inputs
    fn outputs(&self) -> uint;  // number of outputs

    // get and set summary information as part of set-up.
    fn get_internal_summary(&mut self) -> (Vec<Vec<Antichain<S>>>,
                                           Vec<Vec<(T, i64)>>) -> ();
    fn set_external_summary(&mut self, summaries: Vec<Vec<Antichain<S>>>,
                                       external: &Vec<Vec<(T, i64)>>) -> ();

    // push and pull progress information at run-time.
    fn push_external_progress(&mut self, external: &Vec<Vec<(T, i64)>>) -> ();
    fn pull_internal_progress(&mut self, internal: &mut Vec<Vec<(T, i64)>>,
                                         consumed: &mut Vec<Vec<(T, i64)>>,
                                         produced: &mut Vec<Vec<(T, i64)>>) -> bool;
}

This definition is something of a mouthful, but I hope by the end of the post the reasons for each of the parts will become clearer.

Inputs and outputs

Each scope is, to the outside world, a vertex that consumes and produces messages timestamped by T. The first thing the outside world will need to know is the shape of the scope: how many inputs does it have, and how many outputs does it have. While not conceptually deep, all further communication between the scope and the outer world will be in terms of these inputs and outputs.

Initialization

Two methods are used as part of initializing the computation, the scope summarizing itself to the outer world, and the outer world summarizing itself back to the scope. Because of the hierarchical structure, scopes first summarize themselves to their parent scope, who eventually has enough information to summarize the structure around the scope back to the scope itself.

These summaries include both what the scope (and the outer world) are capable of doing to messages received at their inputs in terms of messages produced at their outputs, and also any initial counts of timestamped messages at each output.

The first method is get_internal_summary, which asks the scope for path summaries for each of its (input, output) pairs, as well as initial counts at each of its outputs for each timestamp.

fn get_internal_summary(&mut self) -> (Vec<Vec<Antichain<S>>>,
                                       Vec<Vec<(T, i64)>>) -> ();

The result is simply the pair of summaries for each input to each output, and the list for each output of increments to counts for each timestamp.

The second method is set_internal_summary, which reports back to the scope the path summaries from each of its outputs back to its inputs, as well as the initial counts for each of its inputs.

fn set_external_summary(&mut self, summaries: Vec<Vec<Antichain<S>>>,
                                   external: &Vec<Vec<(T, i64)>>) -> ();

The two arguments are the counterparts to the returned values above: path summaries from each of the scope’s outputs to each of the scope’s inputs, and initial counts at each of the scope’s inputs.

Runtime

Once the computation has begun, the outer world interacts with the scope in two ways: it notifies the scope of progress made in the outer world, and it requests information about progress made in the scope itself. These are the push_external_progress and pull_internal_progress methods.

The first of these two methods is relatively simple: it communicates to the scope a list of updates to timestamp counts for each of its inputs. These updates communicates progress in the outer world projected down to the scope’s inputs.

fn push_external_progress(&mut self, external: &Vec<Vec<(T, i64)>>) -> ();

The method does not return anything, reflecting our desire to have this be asynchronous communication: the notification is not coupled with a response, it is simply sent to the scope from the outer world with the hope that progress will be made in the future.

The second method is slightly more complicated, at least in signature. Its role is to retrieve information from the scope about progress it has made, so that the system as a whole may advance.

fn pull_internal_progress(&mut self, internal: &mut Vec<Vec<(T, i64)>>,
                                     consumed: &mut Vec<Vec<(T, i64)>>,
                                     produced: &mut Vec<Vec<(T, i64)>>) -> bool;

The arguments are vectors meant to be populated, indicating (respectively) the changes to any internal work projected to the scope’s outputs, the numbers of messages consumed at each of the scope’s inputs, and the numbers of messages produced at each of the scope’s outputs. The method returns a bool indicating whether there is unreported work that may not result in output messages that should prevent the computation from shutting down (for example, vertices with side-effects but no outputs).

This method collects three types of updates together because it is important for correctness that two types of updates occur together:

Consuming an input message resulting in more internal work (consuming -> internal).
Producing an output message resulting in less internal work (internal -> producing).

If either of these two become decoupled, it is possible the outer world may have an incomplete (and incorrect) view of the state of the scope. For example, if the scope acknowledges consuming an input message but does not indicate the new internal work, the outer scope may reasonably assume there is none, and communicate incorrect progress information to others. Likewise, if internal work is retired producing an output message, there is no reason to believe the outside world will know about the message in a timely fashion unless told by the scope itself.

While we might have broken the above into two pull_internal methods, there is a nice symmetry in having one method for each direction of information flow. Moreover, there does not yet seem to be an advantage in soliciting only a subset of progress information from a scope. The scope can report what it wants to report.

Next steps

We’ve described the interfaces currently in place for our hierarchical implementation of timely dataflow. There is still a lot to say about how to implement scopes, especially the generic logic for a scope which wraps a connected collection of other scopes. We’ll talk about this next, and almost certainly surface details that are still up for discussion.