[TableGen] Introduce a generic automaton (DFA) backend

Summary:
This patch introduces -gen-automata, a backend for generating deterministic finite-state automata.

DFAs are already generated by the -gen-dfa-packetizer backend. This backend is more generic and will
hopefully be used to implement the DFA generation (and determinization) for the packetizer in the
future.

This backend allows not only generation of a DFA from an NFA (nondeterministic finite-state
automaton), it also emits sidetables that allow a path through the DFA under a sequence of inputs to
be analyzed, and the equivalent set of all possible NFA transitions extracted.

This allows a user to not just answer "can my problem be solved?" but also "what is the
solution?". Clearly this analysis is more expensive than just playing a DFA forwards so is
opt-in. The DFAPacketizer has this behaviour already but this is a more compact and generic
representation.

Examples are bundled in unittests/TableGen/Automata.td. Some are trivial, but the BinPacking example
is a stripped-down version of the original target problem I set out to solve, where we pack values
(actually immediates) into bins (an immediate pool in a VLIW bundle) subject to a set of esoteric
constraints.

Reviewers: t.p.northover

Subscribers: mgorny, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D67968

llvm-svn: 373718

Author: James Molloy

llvm/include/llvm/Support/Automaton.h

@@ -0,0 +1,230 @@
+//===-- Automaton.h - Support for driving TableGen-produced DFAs ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements class that drive and introspect deterministic finite-
+// state automata (DFAs) as generated by TableGen's -gen-automata backend.
+//
+// For a description of how to define an automaton, see
+// include/llvm/TableGen/Automaton.td.
+//
+// One important detail is that these deterministic automata are created from
+// (potentially) nondeterministic definitions. Therefore a unique sequence of
+// input symbols will produce one path through the DFA but multiple paths
+// through the original NFA. An automaton by default only returns "accepted" or
+// "not accepted", but frequently we want to analyze what NFA path was taken.
+// Finding a path through the NFA states that results in a DFA state can help
+// answer *what* the solution to a problem was, not just that there exists a
+// solution.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_AUTOMATON_H
+#define LLVM_SUPPORT_AUTOMATON_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Support/Allocator.h"
+#include <deque>
+#include <map>
+#include <memory>
+#include <unordered_map>
+#include <vector>
+
+namespace llvm {
+
+using NfaPath = SmallVector<uint64_t, 4>;
+
+/// Forward define the pair type used by the automata transition info tables.
+///
+/// Experimental results with large tables have shown a significant (multiple
+/// orders of magnitude) parsing speedup by using a custom struct here with a
+/// trivial constructor rather than std::pair<uint64_t, uint64_t>.
+struct NfaStatePair {
+  uint64_t FromDfaState, ToDfaState;
+
+  bool operator<(const NfaStatePair &Other) const {
+    return std::make_tuple(FromDfaState, ToDfaState) <
+           std::make_tuple(Other.FromDfaState, Other.ToDfaState);
+  }
+};
+
+namespace internal {
+/// The internal class that maintains all possible paths through an NFA based
+/// on a path through the DFA.
+class NfaTranscriber {
+private:
+  /// Cached transition table. This is a table of NfaStatePairs that contains
+  /// zero-terminated sequences pointed to by DFA transitions.
+  ArrayRef<NfaStatePair> TransitionInfo;
+
+  /// A simple linked-list of traversed states that can have a shared tail. The
+  /// traversed path is stored in reverse order with the latest state as the
+  /// head.
+  struct PathSegment {
+    uint64_t State;
+    PathSegment *Tail;
+  };
+
+  /// We allocate segment objects frequently. Allocate them upfront and dispose
+  /// at the end of a traversal rather than hammering the system allocator.
+  SpecificBumpPtrAllocator<PathSegment> Allocator;
+
+  /// Heads of each tracked path. These are not ordered.
+  std::deque<PathSegment *> Heads;
+
+  /// The returned paths. This is populated during getPaths.
+  SmallVector<NfaPath, 4> Paths;
+
+  /// Create a new segment and return it.
+  PathSegment *makePathSegment(uint64_t State, PathSegment *Tail) {
+    PathSegment *P = Allocator.Allocate();
+    *P = {State, Tail};
+    return P;
+  }
+
+  /// Pairs defines a sequence of possible NFA transitions for a single DFA
+  /// transition.
+  void transition(ArrayRef<NfaStatePair> Pairs) {
+    // Iterate over all existing heads. We will mutate the Heads deque during
+    // iteration.
+    unsigned NumHeads = Heads.size();
+    for (auto HeadI = Heads.begin(), HeadE = std::next(Heads.begin(), NumHeads);
+         HeadI != HeadE; ++HeadI) {
+      PathSegment *Head = *HeadI;
+      // The sequence of pairs is sorted. Select the set of pairs that
+      // transition from the current head state.
+      auto PI = lower_bound(Pairs, NfaStatePair{Head->State, 0ULL});
+      auto PE = upper_bound(Pairs, NfaStatePair{Head->State, INT64_MAX});
+      // For every transition from the current head state, add a new path
+      // segment.
+      for (; PI != PE; ++PI)
+        if (PI->FromDfaState == Head->State)
+          Heads.push_back(makePathSegment(PI->ToDfaState, Head));
+    }
+    // Now we've iterated over all the initial heads and added new ones,
+    // dispose of the original heads.
+    Heads.erase(Heads.begin(), std::next(Heads.begin(), NumHeads));
+  }
+
+public:
+  NfaTranscriber(ArrayRef<NfaStatePair> TransitionInfo)
+      : TransitionInfo(TransitionInfo) {
+    reset();
+  }
+
+  void reset() {
+    Paths.clear();
+    Heads.clear();
+    Allocator.DestroyAll();
+    // The initial NFA state is 0.
+    Heads.push_back(makePathSegment(0ULL, nullptr));
+  }
+
+  void transition(unsigned TransitionInfoIdx) {
+    unsigned EndIdx = TransitionInfoIdx;
+    while (TransitionInfo[EndIdx].ToDfaState != 0)
+      ++EndIdx;
+    ArrayRef<NfaStatePair> Pairs(&TransitionInfo[TransitionInfoIdx],
+                                 EndIdx - TransitionInfoIdx);
+    transition(Pairs);
+  }
+
+  ArrayRef<NfaPath> getPaths() {
+    Paths.clear();
+    for (auto *Head : Heads) {
+      NfaPath P;
+      while (Head->State != 0) {
+        P.push_back(Head->State);
+        Head = Head->Tail;
+      }
+      std::reverse(P.begin(), P.end());
+      Paths.push_back(std::move(P));
+    }
+    return Paths;
+  }
+};
+} // namespace internal
+
+/// A deterministic finite-state automaton. The automaton is defined in
+/// TableGen; this object drives an automaton defined by tblgen-emitted tables.
+///
+/// An automaton accepts a sequence of input tokens ("actions"). This class is
+/// templated on the type of these actions.
+template <typename ActionT> class Automaton {
+  /// Map from {State, Action} to {NewState, TransitionInfoIdx}.
+  /// TransitionInfoIdx is used by the DfaTranscriber to analyze the transition.
+  /// FIXME: This uses a std::map because ActionT can be a pair type including
+  /// an enum. In particular DenseMapInfo<ActionT> must be defined to use
+  /// DenseMap here.
+  std::map<std::pair<uint64_t, ActionT>, std::pair<uint64_t, unsigned>> M;
+  /// An optional transcription object. This uses much more state than simply
+  /// traversing the DFA for acceptance, so is heap allocated.
+  std::unique_ptr<internal::NfaTranscriber> Transcriber;
+  /// The initial DFA state is 1.
+  uint64_t State = 1;
+
+public:
+  /// Create an automaton.
+  /// \param Transitions The Transitions table as created by TableGen. Note that
+  ///                    because the action type differs per automaton, the
+  ///                    table type is templated as ArrayRef<InfoT>.
+  /// \param TranscriptionTable The TransitionInfo table as created by TableGen.
+  ///
+  /// Providing the TranscriptionTable argument as non-empty will enable the
+  /// use of transcription, which analyzes the possible paths in the original
+  /// NFA taken by the DFA. NOTE: This is substantially more work than simply
+  /// driving the DFA, so unless you require the getPaths() method leave this
+  /// empty.
+  template <typename InfoT>
+  Automaton(ArrayRef<InfoT> Transitions,
+            ArrayRef<NfaStatePair> TranscriptionTable = {}) {
+    if (!TranscriptionTable.empty())
+      Transcriber =
+          std::make_unique<internal::NfaTranscriber>(TranscriptionTable);
+    for (const auto &I : Transitions)
+      // Greedily read and cache the transition table.
+      M.emplace(std::make_pair(I.FromDfaState, I.Action),
+                std::make_pair(I.ToDfaState, I.InfoIdx));
+  }
+
+  /// Reset the automaton to its initial state.
+  void reset() {
+    State = 1;
+    if (Transcriber)
+      Transcriber->reset();
+  }
+
+  /// Transition the automaton based on input symbol A. Return true if the
+  /// automaton transitioned to a valid state, false if the automaton
+  /// transitioned to an invalid state.
+  ///
+  /// If this function returns false, all methods are undefined until reset() is
+  /// called.
+  bool add(const ActionT &A) {
+    auto I = M.find({State, A});
+    if (I == M.end())
+      return false;
+    if (Transcriber)
+      Transcriber->transition(I->second.second);
+    State = I->second.first;
+    return true;
+  }
+
+  /// Obtain a set of possible paths through the input nondeterministic
+  /// automaton that could be obtained from the sequence of input actions
+  /// presented to this deterministic automaton.
+  ArrayRef<NfaPath> getNfaPaths() {
+    assert(Transcriber && "Can only obtain NFA paths if transcribing!");
+    return Transcriber->getPaths();
+  }
+};
+
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_AUTOMATON_H

llvm/include/llvm/TableGen/Automaton.td

@@ -0,0 +1,95 @@
+//===- Automaton.td ----------------------------------------*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the key top-level classes needed to produce a reasonably
+// generic finite-state automaton.
+//
+//===----------------------------------------------------------------------===//
+
+// Define a record inheriting from GenericAutomaton to generate a reasonably
+// generic finite-state automaton over a set of actions and states.
+//
+// This automaton is defined by:
+//   1) a state space (explicit, always bits<32>).
+//   2) a set of input symbols (actions, explicit) and
+//   3) a transition function from state + action -> state.
+//
+// A theoretical automaton is defined by <Q, S, d, q0, F>:
+//   Q: A set of possible states.
+//   S: (sigma) The input alphabet.
+//   d: (delta) The transition function f(q in Q, s in S) -> q' in Q.
+//   F: The set of final (accepting) states.
+//
+// Because generating all possible states is tedious, we instead define the
+// transition function only and crawl all reachable states starting from the
+// initial state with all inputs under all transitions until termination.
+//
+// We define F = S, that is, all valid states are accepting.
+//
+// To ensure the generation of the automaton terminates, the state transitions
+// are defined as a lattice (meaning every transitioned-to state is more
+// specific than the transitioned-from state, for some definition of specificity).
+// Concretely a transition may set one or more bits in the state that were
+// previously zero to one. If any bit was not zero, the transition is invalid.
+//
+// Instead of defining all possible states (which would be cumbersome), the user
+// provides a set of possible Transitions from state A, consuming an input
+// symbol A to state B. The Transition object transforms state A to state B and
+// acts as a predicate. This means the state space can be discovered by crawling
+// all the possible transitions until none are valid.
+//
+// This automaton is considered to be nondeterministic, meaning that multiple
+// transitions can occur from any (state, action) pair. The generated automaton
+// is determinized, meaning that is executes in O(k) time where k is the input
+// sequence length.
+//
+// In addition to a generated automaton that determines if a sequence of inputs
+// is accepted or not, a table is emitted that allows determining a plausible
+// sequence of states traversed to accept that input.
+class GenericAutomaton {
+  // Name of a class that inherits from Transition. All records inheriting from
+  // this class will be considered when constructing the automaton.
+  string TransitionClass;
+
+  // Names of fields within TransitionClass that define the action symbol. This
+  // defines the action as an N-tuple.
+  //
+  // Each symbol field can be of class, int, string or code type.
+  //   If the type of a field is a class, the Record's name is used verbatim
+  //     in C++ and the class name is used as the C++ type name.
+  //   If the type of a field is a string, code or int, that is also used
+  //     verbatim in C++.
+  //
+  // To override the C++ type name for field F, define a field called TypeOf_F.
+  // This should be a string that will be used verbatim in C++.
+  //
+  // As an example, to define a 2-tuple with an enum and a string, one might:
+  //   def MyTransition : Transition {
+  //     MyEnum S1;
+  //     int S2;
+  //   }
+  //   def MyAutomaton : GenericAutomaton }{
+  //     let TransitionClass = "Transition";
+  //     let SymbolFields = ["S1", "S2"];
+  //     let TypeOf_S1 = "MyEnumInCxxKind";
+  //   }
+  list<string> SymbolFields;
+}
+
+// All transitions inherit from Transition.
+class Transition {
+  // A transition S' = T(S) is valid if, for every set bit in NewState, the
+  // corresponding bit in S is clear. That is:
+  //   def T(S):
+  //     S' = S | NewState
+  //     return S' if S' != S else Failure
+  //
+  // The automaton generator uses this property to crawl the set of possible
+  // transitions from a starting state of 0b0.
+  bits<32> NewState;
+}