# Previous Consensus Algorithm

## Introduction

The previous consensus algorithm (the one within Reowolf 1.0 and 1.1) had support for speculative execution. This means that the user may (directly or indirectly) fork the execution of a component. That particular execution then becomes two executions. At some point a component will have to choose which particular execution will be committed to memory. This is one reason for the existence of a `sync` block: a block of code wherein one may perform forking, and at the end a component will have to choose the execution that is committed to memory.

With speculative execution we may have multiple components that are all forking their execution and sending/receiving messages. So we do not end up with one component that has to choose its final execution, but all components choosing their final execution. Note that one component's execution may apply restrictions on the validity of another component's execution. As an example, suppose the following components and their executions:

- Component A: Has two executions:

    - Execution A1: Component A has sent a message to component B.

    - Execution A2: Component A has received a message from component B.

- Component B: Has three executions:

    - Execution B1: Component B has received a message from component A, then sends a message back to component A.

    - Execution B2: Component B has received a message from component A.

    - Execution B3: Component B has sent two messages to component A.

Without delving into too much detail, one may see that the only valid solution to this problem is the combination of `A1` and `B2`.

## Component Execution Tree, and Execution Traces

Components execute PDL code, which may contain calls to `fork`, `put`, and `get`. A `fork` explicitly forks the execution of the code. A `put` sends a message to a particular component, and a `get` receives a message from a component and forks (as explained later).

As the component enters a `sync` block, it has only one possible execution. But as stated above there are reasons for the execution to split up. These individual executions may themselves split up later, thereby forming a so-called "execution tree":

```

                             +-----+       +------+

                             | put |------>| sync |

+-------+      +------+----->+-----+       | end  |

| sync  |      | fork |                    +------+

| start |----->+------+----->+-----+

+-------+                    | get |------>+------+

                             +-----+       | sync |

                                   |       | end  |

                                   |       +------+

                                   |

                                   +------>+------+

                                   |       | sync |

                                   |       | end  |

                                   |       +------+

                                   |

                                   +--> ...

```

This corresponds to the following PDL code:

```

primitive some_component(out<u32> tx, in<u32> rx) {

  sync {

    fork {

      put(tx, 5);

    } or fork {

      get(rx, 1);

    }

}

```

We can see the reason for calling the execution tree a "tree". There are several things to note about the execution tree: Firstly that some executions have been completed and form a complete trace, that is: starting from the "sync start" a complete trace may be represented by the line running to the "sync end". Conversely, there is one trace that is incomplete: there is a trace waiting at the `get` for a message. We'll call a place where the execution splits into multiple branches/executions a "branching point".

Note that the branching points can in the *general case* only be discovered at runtime. Any code may have control flow points like `if` statements, or `while` loops. Consider the following code:

```

primitive some_component(out<u32> tx, bool which_way) {

  sync {

    if (which_way) {

      fork {

        put(tx, 1);

      } or fork {

        put(tx, 2);

      }

    } else {

      put(tx, 3);

    }

  }

}

```

Depending on the value of `which_way` we produce two different execution trees (of which we can determine all traces). The compiler cannot decide at compile time which execution tree will be generated.

Note that the `get` branching points have an arbitrary number of forked executions arising from them. We'll call them "waiting points". In the *general case* we cannot figure out how many forked executions arise from a `get` branching point. The reason being might be illustrated by the following simple example:

```

primitive sender(out<u32> tx, u32 num_forks) {

  sync {

    auto fork_counter = 1;

    while (fork_counter < num_forks) {

      fork {

        put(tx, fork_counter); 

      } or fork { } // empty case

    }

    put(tx, num_forks);

  }

}

primitive receiver(in<u32> rx) {

  u32[] values = {};

  sync {

    bool keep_going = true;

    while (keep_going) {

      auto new_value = get(rx);

      values @= { new_value }; // append

      fork { 

        keep_going = false; 

      } or fork { }

    }

  }

}

```

If the sender is connected to the receiver, then the sender will send anywhere between `1` and `num_forks` messages (distributed over `num_forks` forks), depending on a user-supplied parameter (which we cannot figure out at compile-time). The isolated receiver can generate an infinite number of forked executions. We can analyze that the receiver will at most have `num_forks + 1` forked executions arising from its `get` branching point (the `num_forks` branches that do receive, and one final fork that is infinitely waiting on another message), but the compiler cannot.

For this reason a `get` branching point needs to be kept around for the entire duration of the sync block. The runtime will always need to have a copy of the component's memory and execution state the moment it encountered a `get` instruction, because it might just be that another component (in perhaps a new fork, which we cannot predict) will send it another message, such that it needs to produce a new forked execution.

A `get` operation is also a "blocking operation": in the *general case* the component needs to know the value produced by the `get` operation in order to continue its execution (perhaps more specifically: the first time a `read` operation is performed on the variable that will store the transmitted message). Consider the simple case where the received message contains a boolean that is used in the test expression of an `if` statement: we'll need to have actually received that boolean before we can decide which control flow path to take. Speculating on the contents of messages is too computationally expensive to be taken seriously. A put operation is not a blocking operation: the message is sent and the component continues executing its code.

We've touched upon control flow points multiple times. We'll touch upon some aspects of control flow here, to more easily introduce the algorithm for finding consensus later. A component is fully described by its memory (i.e. all of the memory locations it has access to through its variables) and execution state (i.e. its current position in its code). So once a component encounters a control flow point, it can only take one control flow path. The calling of certain impure functions (e.g. retrieving a cryptographically secure random number) does not change this fact. Note that receiving values from other components might change a component's memory state, hence influence the control flow path it takes in the subsequent forked execution. Conversely, a component sending a value might influence another component's memory state.

So before treading into more detail, here we've found that in the general case:

- A interior of a sync block demarks the place where speculative execution may occur.

- Speculative execution implies that we end up with an execution tree.

- A path through the execution tree that reaches the end of the sync block is called a trace, and represents a valid execution of the sync block for the component (but perhaps not for a peer it interacted with).

- The set of traces produced by a component in its sync block can practically only be discovered at runtime.

- A `get` operation is necessarily a blocking operation that always incurs a branching point. A `put` operation is a nonblocking operation that does not incur a branching point.

- The trace of a component is influenced by the messages it has received.

## Towards a Consensus Algorithm

The solution to the consensus problem is somehow discovering the ways in which the components have influenced the memory state of their peers. If we have a complete trace for each component, for which all peer components agree on the way they have influenced that complete trace, then we've found a solution to the consensus problem. Hence we can subdivide the consensus problem into four parts:

1. Keeping track of the messages that influence the memory state of components.

2. Keeping track of the peers that influence the memory state of components.

3. Finding a set of interactions between components on which all involved components agree, i.e. each `put` should have a corresponding `get` at the peer.

4. Somehow having a communication protocol that finds these agreeable interactions.

We'll incrementally work towards a solution that satisfies the first three points. We'll not consider the last point, as this is essentially a gossip protocol. We define some terms to make the following discussion easier:

- "sync region": The group of components that have interacted with one another and should agree on the global consensus solutionj.

- "local solution": A complete trace of a component. For the component this is a valid local solution, but might not be part of a global solution.

- "global solution": A set of traces, one for each of the components in the sync region, that all agree on the interactions that took place between the components in the sync region.

Suppose a component can somehow predict exactly which messages we're going to receive during the execution of its code, we'll assume that each received message has the appropriate `get` call associated with it. In this case we're able to produce the set of complete traces that a component produces by symbolically executing its code: we start out with the initial memory state, might perhaps do some explicit `fork`ing, know exactly which messages we receive and how they influence the control flow, and arrive at the end of the sync block.

However, as we've outlined above, we cannot know exactly which messages we're going to receive. We'll have to discover these messages while executing a component. The next best thing is to keep track of the values of the messages that we've received in a complete trace. Once we have complete traces for all of the interacting components, we can check that the received value corresponds to a sent value. e.g.

```

primitive sender(out<u32> tx) {

  sync {

    fork {

      put(tx, 1);   

    } or fork {

      put(tx, 2);

    }

  }

}

primitive receiver(in<u32> rx) {

  u32 value = 0; 

  sync {

    value = get(rx);

  }

}

```

Where `tx` is part of the same channel as `rx`. In this case we'll have two traces for each of the components, resulting in two valid global consensus solutions. In one solution the message `1` was transferred, in another the message `2` was transferred. There are two problems with this solution: firstly it doesn't take the identity of the channel into account. And secondly it doesn't take the effects of previous messages into account.

To illustrate the first problem, consider:

```

primitive sender(out<u32> tx_a, out<u32> tx_b) {

  sync {

    fork {

      put(tx_a, 1);

    } or fork {

      put(tx_b, 1);

    }

  }

}

primitive receiver(in<u32> rx_a, in<u32> rx_b) {

  u32 value = 0; 

  sync {

    value = get(rx_a);

  }

}

```

Here the fact that the sender has the solutions `1` and `1` does not help the receiver figure out which of those corresponds to its own solution of `1`.

To illustrate the second problem, consider:

```

primitive sender(out<u32> tx) {

  sync {

    fork {

      put(tx, 1);

      put(tx, 2);

    } or fork {

      put(tx, 2);

    }

  }

}

primitive receiver(in<u32> rx) {

  u32 value = 0; 

  sync {

    value = get(rx);

  }

}

```

Now we'll have `sender` contributing the solutions `1, 2` and `2`. While the receiver will generate the solutions `1`, `2` and `2`. The reason there are three solutions for the receiver is because it cannot figure out that the message `2` from the sender depended on the first message `1` from the sender having arrived.

We can solve this by somehow embedding the identity of the channel associated with the message, and by describing all of the previous interactions

# Known Issues

The current implementation of Reowolf has the following known issues:

- Cannot create uninitialized variables that are later known to be initialized. This is not a problem for the regular types (perhaps a bit tedious), but is a problem for channels/ports. That is to say: if a component needs a temporary variable for a port, then it must create a complete channel. e.g.

  ```

  comp send(out<u32> tx1, out<u32> tx2, in<bool> which) {

    channel unused -> temporary;

    while (true) sync {

      if (get(which)) {

        temporary = tx1;

      } else {

        temporary = tx2;

      }

      put(temporary, 1);

    }

  }

  ```

- Reserved memory for ports will grow without bounds: Ports can be given away from one component to another by creating a component, or by sending a message containing them. The component sending those ports cannot remove them from its own memory if there are still other references to the transferred port in its memory. This is because we want to throw a reasonable error if that transferred port is used by the original owner. Hence we need to keep some information about that transferred port in the sending component's memory. The solution is to have reference counting for the ports, but this is not implemented.

- An extra to the above statements: when transferring ports to a new component, the memory that remembers the state of that port is removed from the component that is creating the new one. Hence using old references to that port within the creating component's PDL code results in a crash.

- Some control algorithms are not robust under multithreading. Mainly error handling when in sync mode (because there needs to be a revision where we keep track of which components are still reachable by another component). And complicated scenarios where ports are transferred.

    Component,

    CompTcpListener,

/// Generic storage for functions and components.

    ComponentTcpListener,

            ComponentRandomU32 | ComponentTcpClient | ComponentTcpListener => true,

                                Method::ComponentRandomU32 | Method::ComponentTcpClient | Method::ComponentTcpListener => {

    /// Again a somewhat temporary method. Can be used by components to look up

    /// the definition of a particular procedure. Intended use is to find the

    /// DefinitionId/TypeId of builtin components.

    pub(crate) fn find_procedure(&self, module_name: &[u8], proc_name: &[u8]) -> Option<(ProcedureDefinitionId, TypeId)> {

        // Lookup type definition in module

        let root_id = self.lookup_module_root(module_name)?;

        let module = &self.heap[root_id];

        let definition_id = module.get_definition_by_ident(&self.heap, proc_name)?;

        let definition = &self.heap[definition_id];

        // Make sure the procedure is not polymorphic

        if !definition.poly_vars().is_empty() {

            return None;

        }

        if !definition.is_procedure() {

            return None;

        }

        // Lookup in type table

        let definition = definition.as_procedure();

        let type_parts = [ConcreteTypePart::Component(definition.this, 0)];

        let type_id = self.types.get_monomorph_type_id(&definition.this.upcast(), &type_parts)

            .expect("type ID for non-polymorphic procedure");

        return Some((definition.this, type_id));

    }

    pub fn as_struct(&'a self) -> StructTypeInspector<'a> {

        let heap = self.heap.as_struct();

        let type_table = self.type_table.variant.as_struct();

        return StructTypeInspector{ heap, type_table };

    }

}

pub struct StructTypeInspector<'a> {

    heap: &'a StructDefinition,

    type_table: &'a StructMonomorph,

}

impl StructTypeInspector<'_> {

    /// Retrieves number of struct fields

    pub fn get_num_struct_fields(&self) -> usize {

        return self.heap.fields.len();

    }

    /// Retrieves struct field index

    pub fn get_struct_field_index(&self, field_name: &[u8]) -> Option<usize> {

        let field_index = self.heap.fields.iter()

            .position(|v| v.field.value.as_bytes() == field_name)?;

        return Some(field_index);

    }

                Some(KW_COMPONENT) => self.visit_component_definition(module, &mut iter, ctx)?,

        let (_component_text, _) = consume_any_ident(&module.source, iter)?;

        debug_assert!(_component_text == KW_COMPONENT);

                ("std.internet", "tcp_listener") => ProcedureSource::CompTcpListener,

                ProcedureKind::Component =>

                                    // Bit of a hack, at this point the source is not yet known, except if it is a

                                    // builtin. So we check for the "kind"

                                    ProcedureSource::FuncUserDefined | ProcedureSource::CompUserDefined => {

                                        match proc_def.kind {

                                            ProcedureKind::Function => Method::UserFunction,

                                            ProcedureKind::Component => Method::UserComponent,

                                        }

                                    },

                                    ProcedureSource::CompTcpListener => Method::ComponentTcpListener,

            KW_COMPONENT => {

                    ProcedureDefinition::new_empty(this, module.root_id, ProcedureKind::Component, identifier, poly_vars)

            ident == KW_COMPONENT

        if self.proc_kind != ProcedureKind::Component {

                "synchronous statements may only be used in components"

        if self.proc_kind != ProcedureKind::Component {

                "select statements may only be used in components"

        // Make sure the new statement occurs inside a component

        if self.proc_kind != ProcedureKind::Component {

                "instantiating components may only be done in components"

            | Method::ComponentTcpClient

            | Method::ComponentTcpListener => {

pub(crate) const KW_COMPONENT: &'static [u8] = b"comp";

        KW_STRUCT | KW_ENUM | KW_UNION | KW_FUNCTION | KW_COMPONENT => true,

    pub(crate) fn as_struct(&self) -> &StructMonomorph {

        match self {

            MonoTypeVariant::Struct(v) => v,

            _ => unreachable!(),

        }

    }

        comp f() {

        "comp f() { sync select {} }"

        comp f() {