Merge pull request #43 from RisingStack/master

feat: replace api key with personal access token

Author: celo0213

Dockerfile

@@ -7,6 +7,6 @@ COPY . .
 
 RUN npm i --production
 
-EXPOSE 3001
+EXPOSE 8080
 
 CMD ["node", "."]

README.md

@@ -35,23 +35,91 @@ $ npm start
 
 ## For some basic source code explanation see the [wiki page](https://github.com/nditech/CyberSim-Backend/wiki)
 
-## AWS environment:
+# CyberSim-Backend Deployment Guide
 
-### The AWS environment supports both continuous integration and continuous deployment. The environment is made of the following components:
+- The CyberSim Game comprises two distinct applications: a Node.js-based backend API and a React-based frontend UI. This guide specifically covers the deployment process for the Node.js-based backend API. For instructions on deploying the frontend application, please refer to the [CyberSim-UI README](https://github.com/nditech/CyberSim-UI#readme).
 
-- **Github repository (nditech/CyberSim-Backend)**: Each change on the local repositories are pushed to a new branch in the Github remote. Once these changes are reviewed, they are merged into the `master` branch.
+1. [Set up the Elastic Beanstalk environment](#elastic-beanstalk-eg-ndicybersimgame-env)
+2. [Set up the CodePipeline](#codepipeline-eg-ndicybersim-backend)
 
-- **CodePipeline (ndi-cybersim-backend-staging, ndi-cybersim-backend-prod)**: A CodePipeline project is created for both staging and production environments. For the staging environment a webhook is registered on Github, so each change on the `master` branch will trigger and automatic build on AWS. For the production environment no webhook is registered so changes on the `master` branch will require a manual release in CodePipeline.
+## Environment Component Naming Convention
 
-- **CodeBuild (ndi-cybersim-backend)**: A single CodeBuild project is created to build and test both staging and production changes. After a change is triggered on CodePiepeline (manually or automatically), the source code is transferred to CodeBuild and the steps defined in the `buildspec.yml` file are executed. These steps include the installation of node, npm packages and Postgres on a virtual machine and the execution of predefined database related tests. Once these tests are run, CodePiepeline begins the deployment.
+The environment component name follows this format: `<ACCOUNT_ALIAS>@<COMPONENT_NAME>`.
 
-- **Elastic Beanstalk (NdiCybersimBackend-staging, NdiCybersimBackend-prod)**: In Elastic Beanstalk (EB) a different environment is created for both staging and production. In both environments, the node application is running inside a docker container so an existing Dockerfile inside the source code is necessary for EB. The docker image is created using the Dockerfile in the root. Once the deployment is compleated, the API will be live. For EB the following environment variables must be set (These variables can be set seberatly for each EB environment under the Configuration/Software Tab of the AWS Console.):
+## GitHub Repository (nditech/CyberSim-UI)
 
-  - **PORT**: The PORT must match the port exposed in the docker container which currently is **3001**.
-  - **NODE_ENV**: Must be either `production`, `development` or `test`. If the given value is `test` the server will reset the Postgres Database on each restart.
-  - **DB_URL**: The connection string for the Postgres Database which is the following: postgres://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DB_NAME>
+All local repository changes are pushed to new branches in the GitHub remote repository. These changes are reviewed and merged into the 'master' branch.
 
-- **RDS (aa1p6s0h1so0mf9)**: A Postgres Database is created to store the data for both staging and production environments The default name of the RDS database is **ebdb**. This DB is only available for the EC2 instance running inside the EB environment. In order to connect to the DB from a different client, port forwarding must be configured. In order to create an ssh tunnel and forward the traffic from the DB to locahost run to following commands:
+## CodePipeline (e.g., ndi@Cybersim-backend)
+
+A separate CodePipeline project is created for each production environment. The pipeline consists of 2 stages:
+
+### SOURCE
+
+1. Set the _Source provider_ to "GitHub (Version 2)" and connect to the following repository: [nditech/CyberSim-Backend](https://github.com/nditech/CyberSim-Backend). Branch name should be `master`.
+2. Keep the _Start the pipeline on source code change_ option checked to trigger automatic builds on AWS whenever changes are pushed to the `master` branch. (Troubleshooting: Manually trigger a release in CodePipeline)
+3. Leave the _Output artifact format_ on the default "CodePipeline default".
+
+### BUILD
+
+Skip the build stage. It's not needed.
+
+### DEPLOY
+
+1. Set the _Deploy provider_ to "AWS Elastic Beanstalk".
+2. [Select the Cybersim application and the environment you've just created.](#elastic-beanstalk-eg-ndicybersimgame-env)
+
+## Elastic Beanstalk (e.g., ndi@Cybersimgame-env)
+
+In Elastic Beanstalk (EB) a different environment is created for each game instance. In each environment, the node application is running inside a docker container so an existing Dockerfile inside the source code is necessary for EB. The docker image is created using the Dockerfile in the root. Once the deployment is compleated, the API will be live.
+
+To create a new Elastic Beanstalk environment follow these steps:
+
+1. For _Environment tier_ select "Web server environment".
+2. _Environment name_ and _domain_ are arbitrary. This domain is required in the [frontend setup](https://github.com/nditech/CyberSim-UI#readme).
+3. For _Platform_ select a managed Docker platform. Recommended: "Docker running on 64bit Amazon Linux 2@3.6.1"
+4. You might want to deploy an existing version right away, you can do it under the _Application code_ setting.
+5. Just leave the _Preset_ on "Single instance".
+6. For _Service role_ settings:
+
+- Existing service role --> "aws-elasticbeanstalk-service-role"
+- Add your SSL key for easier access to the EC2 instance. You can create a new "EC2 keypair" under the "EC2 service console / Network & security / Key pairs".
+- EC2 instance profile --> "aws-elasticbeanstalk-ec2-role"
+
+7. VPC is not required.
+8. Set the _Public IP adress_ to "Activated"
+9. For _Database_ select "Enabled" and follow [these steps](#rds-eg-ndiaa1jiteus75zy8h)
+10. For the _Instances_ and _Capacity_ settings leave everything on default.
+11. Set the _Monitoring_ to "Basic".
+12. _Managed updates_ to "false".
+13. For _Rolling updates and deployments_ everything on default.
+14. Under the _Platform software_ settings you must set these 3 environment variables. You can modify these variables later at any time under the Configuration/Software Tab of the AWS Console:
+
+- **PORT**: The PORT must match the port exposed in the docker container which currently is **8080**.
+- **NODE_ENV**: Must be either `production`, `development` or `test`. If the given value is `test` the server will reset the Postgres Database on each restart.
+- **DB_URL**: The connection string for the Postgres Database which is the following: postgres://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DB_NAME>. You can find the <HOST>:<PORT> information about the DB under the "Connectivity & security / Endpoint (/Port)" tab of the databases RDS page.
+
+15. Optional environment variables:
+
+- **MIGRATION_PASSWORD**: Required for creating a migration on the frontends "Migration" page. Defaults to "secret".
+- **LOG_LEVEL**: Only for the `test` NODE_ENV. Defaults to "error".
+
+## RDS (e.g., ndi@aa1jiteus75zy8h)
+
+A Postgres Database is created to store the data for each environment. When creating a DB for an Elastic Beanstalk environment, by default the database created will get a name like this `awseb-e-<Environment ID>-...`. The default name of the RDS database is **ebdb**.
+
+Database settings:
+
+1. _Engine_ - "postgres"
+2. _Version_ - "15.3"
+3. _Instance class_ - "db.t3.micro"
+4. _Storage_ - "20GB"
+5. _Username_ - arbitrary, required in the "DB_URL" environment variable
+6. _Password_ - arbitrary, required in the "DB_URL" environment variable
+7. _Availability_ - "Low"
+8. _Deletion policy_ - "Snapshot"
+
+This DB is only available for the EC2 instance running inside the EB environment. In order to connect to the DB from a different client, port forwarding must be configured. In order to create an ssh tunnel and forward the traffic from the DB to locahost run to following commands:
 
 ```
 # Create the tunnel using ssh. Please note that you need the private key for this command.

index.js

@@ -26,7 +26,7 @@ const checkEnviroment = async () => {
 
 checkEnviroment()
   .then(() => {
-    const port = process.env.PORT || 3001;
+    const port = process.env.PORT || 8080;
     const http = createServer(app);
     createSocket(http);
     const server = http.listen(port, () => {

knexfile.js

@@ -2,5 +2,8 @@ require('dotenv').config();
 
 module.exports = {
   client: 'pg',
-  connection: process.env.DB_URL,
+  connection: {
+    connectionString: process.env.DB_URL,
+    ssl: { rejectUnauthorized: false },
+  },
 };

src/app.js

@@ -71,15 +71,15 @@ app.get('/curveballs', async (req, res) => {
 });
 
 app.post('/migrate', async (req, res) => {
-  const { password, apiKey, tableId } = req.body;
+  const { password, accessToken, tableId } = req.body;
   if (password === config.migrationPassword) {
     try {
-      await migrate(apiKey, tableId);
+      await migrate(accessToken, tableId);
       res.send();
     } catch (err) {
       if (err.error === 'AUTHENTICATION_REQUIRED') {
         res.status(400).send({
-          apiKey: 'Invalid airtable api key',
+          accessToken: 'Invalid airtable access token',
         });
       } else if (err.error === 'NOT_FOUND') {
         res.status(400).send({
@@ -92,6 +92,16 @@ app.post('/migrate', async (req, res) => {
           message: err.message,
           errors,
         });
+      } else if (err.error === 'NOT_AUTHORIZED') {
+        res.status(400).send({
+          validation: true, // Set this to "true" to show it in an alert box on the frontend.
+          message: err.message,
+          errors: [
+            {
+              message: `An authorization error has occurred! Please make sure that the base ID and the required personal access token scope settings are correct!`,
+            },
+          ],
+        });
       } else {
         console.log(500);
         res.status(500).send({

src/util/errors.js

@@ -1,3 +1,9 @@
+function transformSchemaError(schemaError) {
+  return {
+    message: `A schema error occurred when querying the ${schemaError.tableName} table. Please check if the table is set correctly!`,
+  };
+}
+
 function transformValidationError(validationError) {
   return validationError.errors.map((error) => {
     const [id, message] = error.split('.');
@@ -11,7 +17,13 @@ function transformValidationError(validationError) {
 
 function transformValidationErrors(validationErrors) {
   if (Array.isArray(validationErrors)) {
-    return validationErrors.map(transformValidationError).flat();
+    return validationErrors
+      .map((validationError) =>
+        validationError.validation
+          ? transformValidationError(validationError)
+          : transformSchemaError(validationError),
+      )
+      .flat();
   }
   return transformValidationError(validationErrors);
 }

src/util/migrate.js

@@ -48,6 +48,7 @@ function fetchTable(base, tableName) {
         },
         function done(err) {
           if (err) {
+            err.tableName = tableName;
             reject(err);
           } else {
             validate(airtableSchemas[tableName], allFields, tableName)
@@ -73,16 +74,16 @@ function addPartyLocation(locations) {
     : locations?.[0];
 }
 
-async function migrate(apiKey, baseId) {
+async function migrate(accessToken, baseId) {
   // connect to the airtable instance
   Airtable.configure({
     endpointUrl: 'https://api.airtable.com',
-    apiKey,
+    apiKey: accessToken,
   });
 
   const base = Airtable.base(baseId);
 
-  // do a starting "fake" fetch to check if the API key and table id are correct
+  // do a starting "fake" fetch to check if the personal access token and table id are correct
   await fetchTable(base, 'handbook_categories');
 
   // define arrays for junctions tables that must be added at the end of the migration