Add support for converting OpenAPI3 specs to TypeSpec (microsoft#3663)

fix microsoft#3038
This PR updates the `@typespec/openapi3` package to support converting
OpenAPI3 specs to TypeSpec.

## Example usage:
1. `npm install @typespec/openapi3`
2. `npx openapi3-to-tsp compile --output-dir ./tsp-output
/path/to/openapi-yaml-or-json`

## What's supported
- Parse OpenAPI3 specs in yml/json formats (via 3rd party package)
- Generates file namespace based on OpenAPI3 service name
- Populates `@info` decorator with OpenAPI3 service info
- Converts `#/components/schemas` into TypeSpec models/scalars.
- Converts `#/components/parameters` into TypeSpec models/model
properties as appropriate.
- Generates a response model for every operation/statusCode/contentType
combination.
- Operation tags
- Generates TypeSpec operations with routes/Http Method decorators
- Generates docs/extension decorators
- Most schema decorators
- Model inheritance via `allOf`
- Discriminators

## What's not supported (yet)
- auth
- deprecated directive
- combining multiple versions of an OpenAPI3-defined service into a
single TypeSpec project
- converting `#/components/requestBodies` and `#/components/responses`
into models - TypeSpec doesn't seem to generate these and I didn't find
examples in the wild where they were defined _and_ actually used so
deprioritized.
- emitting warnings/FIXMEs for unexpected/unsupported scenarios
- Probably a lot more that I'm still discovering

## Notes

When going through the TypeSpec -> OpenAPI3 -> TypeSpec loop, the
generated TypeSpec is going to be larger than the original. The biggest
contribution towards this is because I'm currently generating a model
for every possible response on every operation.

I can definitely pare this down with some simple heuristics that take
into account what default statusCode/contentTypes are, and extract the
referenced body type directly in the operation's return signature. I can
also eliminate the `@get` decorators, `@route("/")` routes, and likely
use some of the response models provided by TypeSpec.Http.

However - if I'm using this tool to convert from OpenAPI3 to TypeSpec -
I thought it might be preferable to be more explicit in the generated
output so there's no mystery on how things actually get defined. Will be
interested in feedback on this.


## Testing
For tests, I generate TypeSpec files for a number of OpenAPI3 specs.
Most of the OpenAPI3 specs I generated from our TypeSpec samples
packages. Then I'm able to compare the generated TypeSpec to the
corresponding original TypeSpec file. I've also been diffing the
OpenAPI3 specs generated from the original and generated TypeSpec files
<- these are what typically show no changes outside of known unsupported
conversions (e.g. auth).

---------

Co-authored-by: Christopher Radek <Christopher.Radek@microsoft.com>

Author: chrisradek

.chronus/changes/oa3-to-ts-2024-5-26-13-27-36.md

@@ -0,0 +1,7 @@
+---
+changeKind: feature
+packages:
+  - "@typespec/openapi3"
+---
+
+Adds support for converting OpenAPI3 specs to TypeSpec via the new tsp-openapi3 CLI included in the `@typespec/openapi3` package.

docs/emitters/openapi3/cli.md

@@ -0,0 +1,224 @@
+---
+title: OpenAPI3 to TypeSpec
+---
+
+# tsp-openapi3 CLI
+
+## Converting OpenAPI 3 into TypeSpec
+
+This package includes the `tsp-openapi3` CLI for converting OpenAPI 3 specs into TypeSpec.
+The generated TypeSpec depends on the `@typespec/http`, `@typespec/openapi` and `@typespec/openapi3` libraries.
+
+### Usage
+
+1. via the command line
+
+```bash
+tsp-openapi3 ./openapi3spec.yml --output-dir ./tsp-output
+```
+
+### tsp-openapi3 arguments
+
+The path to the OpenAPI3 yaml or json file **must** be passed as a position argument.
+
+The named arguments are:
+
+| Name       | Type    | Required | Description                                                                              |
+| ---------- | ------- | -------- | ---------------------------------------------------------------------------------------- |
+| output-dir | string  | required | The output directory for generated TypeSpec files. Will be created if it does not exist. |
+| help       | boolean | optional | Show help.                                                                               |
+
+## Examples
+
+### 1. Convert component schemas into models
+
+All schemas present at `#/components/schemas` will be converted into a model or scalar as appropriate.
+
+<table>
+<tr>
+<td>OpenAPI3</td>
+<td>TypeSpec</td>
+</tr>
+<!-- ---------------------------------------------------  SCENARIO 1.1  ----------------------------------------------------------- -->
+<tr>
+<td>
+
+```yml
+components:
+  schemas:
+    Widget:
+      type: object
+      required:
+        - id
+        - weight
+        - color
+      properties:
+        id:
+          type: string
+        weight:
+          type: integer
+          format: int32
+        color:
+          type: string
+          enum:
+            - red
+            - blue
+    uuid:
+      type: string
+      format: uuid
+```
+
+</td>
+<td>
+
+```tsp
+model Widget {
+  id: string;
+  weight: int32;
+  color: "red" | "blue";
+}
+
+@format("uuid")
+scalar uuid extends string;
+```
+
+</td>
+</tr>
+</table>
+
+### 2. Convert component parameters into models or fields
+
+All parameters present at `#/components/parameters` will be converted to a field in a model. If the model doesn't exist in `#/components/schemas`, then it will be created.
+
+<table>
+<tr>
+  <td>OpenAPI3</td>
+  <td>TypeSpec</td>
+</tr>
+<!-- ---------------------------------------------------  SCENARIO 2.1  ----------------------------------------------------------- -->
+<tr>
+<td>
+
+```yml
+components:
+  parameters:
+    Widget.id:
+      name: id
+      in: path
+      required: true
+      schema:
+        type: string
+  schemas:
+    Widget:
+      type: object
+      required:
+        - id
+        - weight
+        - color
+      properties:
+        id:
+          type: string
+        weight:
+          type: integer
+          format: int32
+        color:
+          type: string
+          enum:
+            - red
+            - blue
+```
+
+</td>
+<td>
+
+```tsp
+model Widget {
+  @path id: string;
+  weight: int32;
+  color: "red" | "blue";
+}
+```
+
+</td>
+</tr>
+<!-- ---------------------------------------------------  SCENARIO 2.2  ----------------------------------------------------------- -->
+<tr>
+<td>
+
+```yml
+components:
+  parameters:
+    Foo.id:
+      name: id
+      in: path
+      required: true
+      schema:
+        type: string
+```
+
+</td>
+<td>
+
+```tsp
+model Foo {
+  @path id: string;
+}
+```
+
+</td>
+</tr>
+</table>
+
+### 3. Convert path routes to operations
+
+All routes using one of the HTTP methods supported by `@typespec/http` will be converted into operations at the file namespace level. A model is also generated for each operation response.
+
+At this time, no automatic operation grouping under interfaces is performed.
+
+<table>
+<tr>
+  <td>OpenAPI3</td>
+  <td>TypeSpec</td>
+</tr>
+<!-- ---------------------------------------------------  SCENARIO 3.1  ----------------------------------------------------------- -->
+<tr>
+<td>
+
+```yml
+paths:
+  /{id}:
+    get:
+      operationId: readWidget
+      parameters:
+        - name: id
+          in: path
+          required: true
+          schema:
+            type: string
+      responses:
+        "200":
+          description: The request has succeeded.
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/Widget"
+```
+
+</td>
+<td>
+
+```tsp
+/**
+ * The request has succeeded.
+ */
+model readWidget200ApplicationJsonResponse {
+  @statusCode statusCode: 200;
+  @bodyRoot body: Widget;
+}
+
+@route("/{id}") @get op readWidget(@path id: string): readWidget200ApplicationJsonResponse;
+```
+
+</td>
+</tr>
+</table>

docs/emitters/openapi3/reference/index.mdx

@@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem';
 
 # Overview
 
-TypeSpec library for emitting OpenAPI 3.0 from the TypeSpec REST protocol binding
+TypeSpec library for emitting OpenAPI 3.0 from the TypeSpec REST protocol binding and converting OpenAPI3 to TypeSpec
 
 ## Install
 

packages/openapi3/README.md

@@ -1,6 +1,6 @@
 # @typespec/openapi3
 
-TypeSpec library for emitting OpenAPI 3.0 from the TypeSpec REST protocol binding
+TypeSpec library for emitting OpenAPI 3.0 from the TypeSpec REST protocol binding and converting OpenAPI3 to TypeSpec
 
 ## Install
 

packages/openapi3/cmd/tsp-openapi3.js

@@ -0,0 +1,9 @@
+#!/usr/bin/env node
+import { main } from "../dist/src/cli/cli.js";
+
+main().catch((error) => {
+  // eslint-disable-next-line no-console
+  console.error(error);
+
+  process.exit(1);
+});

packages/openapi3/package.json

@@ -2,7 +2,7 @@
   "name": "@typespec/openapi3",
   "version": "0.57.0",
   "author": "Microsoft Corporation",
-  "description": "TypeSpec library for emitting OpenAPI 3.0 from the TypeSpec REST protocol binding",
+  "description": "TypeSpec library for emitting OpenAPI 3.0 from the TypeSpec REST protocol binding and converting OpenAPI3 to TypeSpec",
   "homepage": "https://typespec.io",
   "readme": "https://github.com/microsoft/typespec/blob/main/README.md",
   "license": "MIT",
@@ -16,6 +16,9 @@
   "keywords": [
     "typespec"
   ],
+  "bin": {
+    "tsp-openapi3": "cmd/tsp-openapi3.js"
+  },
   "type": "module",
   "main": "dist/src/index.js",
   "tspMain": "lib/main.tsp",
@@ -34,7 +37,7 @@
   },
   "scripts": {
     "clean": "rimraf ./dist ./temp",
-    "build": "npm run gen-extern-signature && tsc -p . && npm run lint-typespec-library",
+    "build": "npm run gen-version && npm run gen-extern-signature && tsc -p . && npm run lint-typespec-library",
     "watch": "tsc -p . --watch",
     "gen-extern-signature": "tspd --enable-experimental gen-extern-signature .",
     "lint-typespec-library": "tsp compile . --warn-as-error --import @typespec/library-linter --no-emit",
@@ -44,14 +47,17 @@
     "test:ci": "vitest run --coverage --reporter=junit --reporter=default",
     "lint": "eslint . --max-warnings=0",
     "lint:fix": "eslint . --fix",
-    "regen-docs": "tspd doc .  --enable-experimental  --output-dir ../../docs/emitters/openapi3/reference"
+    "regen-docs": "tspd doc .  --enable-experimental  --output-dir ../../docs/emitters/openapi3/reference",
+    "regen-specs": "cross-env RECORD=true vitest run",
+    "gen-version": "node scripts/generate-version.js"
   },
   "files": [
     "lib/*.tsp",
     "dist/**",
     "!dist/test/**"
   ],
   "dependencies": {
+    "@readme/openapi-parser": "~2.6.0",
     "yaml": "~2.4.5"
   },
   "peerDependencies": {
@@ -62,6 +68,7 @@
   },
   "devDependencies": {
     "@types/node": "~18.11.19",
+    "@types/yargs": "~17.0.32",
     "@typespec/compiler": "workspace:~",
     "@typespec/http": "workspace:~",
     "@typespec/library-linter": "workspace:~",
@@ -72,6 +79,7 @@
     "@vitest/coverage-v8": "^1.6.0",
     "@vitest/ui": "^1.6.0",
     "c8": "^10.1.2",
+    "cross-env": "~7.0.3",
     "rimraf": "~5.0.7",
     "typescript": "~5.5.3",
     "vitest": "^1.6.0"

packages/openapi3/scripts/generate-version.js

@@ -0,0 +1,27 @@
+// @ts-check
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "fs";
+import { join } from "path";
+import { fileURLToPath } from "url";
+const root = fileURLToPath(new URL("..", import.meta.url).href);
+const distDir = join(root, "dist");
+const versionTarget = join(distDir, "version.js");
+
+function loadPackageJson() {
+  const packageJsonPath = join(root, "package.json");
+  return JSON.parse(readFileSync(packageJsonPath, "utf-8"));
+}
+
+function main() {
+  const pkg = loadPackageJson();
+
+  const version = pkg.version;
+
+  if (!existsSync(distDir)) {
+    mkdirSync(distDir, { recursive: true });
+  }
+
+  const versionJs = `export const version = "${version}";`;
+  writeFileSync(versionTarget, versionJs);
+}
+
+main();

packages/openapi3/src/cli/actions/convert/args.ts

@@ -0,0 +1,4 @@
+export interface ConvertCliArgs {
+  "output-dir": string;
+  path: string;
+}

packages/openapi3/src/cli/actions/convert/convert.ts

@@ -0,0 +1,30 @@
+import oaParser from "@readme/openapi-parser";
+import { resolvePath } from "@typespec/compiler";
+import { OpenAPI3Document } from "../../../types.js";
+import { CliHost } from "../../types.js";
+import { handleInternalCompilerError } from "../../utils.js";
+import { ConvertCliArgs } from "./args.js";
+import { generateMain } from "./generators/generate-main.js";
+import { transform } from "./transforms/transforms.js";
+
+export async function convertAction(host: CliHost, args: ConvertCliArgs) {
+  // attempt to read the file
+  const fullPath = resolvePath(process.cwd(), args.path);
+  const model = await parseOpenApiFile(fullPath);
+  const program = transform(model);
+  let mainTsp: string;
+  try {
+    mainTsp = await generateMain(program);
+  } catch (err) {
+    handleInternalCompilerError(err);
+  }
+
+  if (args["output-dir"]) {
+    await host.mkdirp(args["output-dir"]);
+    await host.writeFile(resolvePath(args["output-dir"], "main.tsp"), mainTsp);
+  }
+}
+
+function parseOpenApiFile(path: string): Promise<OpenAPI3Document> {
+  return oaParser.bundle(path) as Promise<OpenAPI3Document>;
+}