TM_ASBUILT_S18b_T01_V1.0Version 1.0

Technical Manual

Software Quality Analysis as a Service (SQUAAD)

Team No.1

Implementers:

Aleksandr Chernousov

Chris Harman

Supicha Phadungslip

Testers:

Kavneet Kaur

Reza Khazali

George Llames

Sahar Pure

04/25/2018

Version History

Date / Author / Version / Changes made / Rationale
04/27/2018 / CH / 1.0 / ●Add technical documentation for application build/deployment / ●Delivery for as-built package

Table of Contents

Technical Manual

Version History

Table of Contents

Build Dependencies

System Requirements

Additional Recommended

Build steps

Docker Toolbox - Windows

Linux & Mac Build

Auto Reload

Generating Docs

Deploying on AWS

Check Application Server Logs

Clearing Out Docker Images to Regain Space (Optionally Run This After Deploy)

HTTPS Cert Generation

Clear Your Database and Rebuild

Populating the DB with New Records

1

TM_ASBUILT_S18b_T01_V1.0Version Date: 04/27/2018

TM_ASBUILT_S18b_T01_V1.0Version 1.0

Build Dependencies

  • Docker Toolbox- for Windows or Mac (ships with everything you will need)
  • Install Virtualbox:brew cask install virtualbox- for Mac Only

or

  • Docker
  • Docker Compose

System Requirements

  • 2GB of RAM
  • 8GB Available Disk Space

Additional Recommended

  • Git Bash- for Windows

If you receive the following error on Windows:No such file or directory bash, it means that your local git pull is converting line endings to Windows formatting CRLF, rather than respecting the line endings in the repo. You can resolve the issue by following the 4 commandshere.

Install the above dependencies before proceeding onto the build steps

Build steps

Docker Toolbox - Windows

(using Git Bash terminal)

Note:Steps 1-2 are required for only the first build - start with step 3 after you have created a Docker Machine successfully.

  1. Confirm Docker toolbox installation was successful by confirming versions of required software. Add your local Docker installation directory to thePATHif the following commands do not work as expected.

$ docker-machine -v

docker-machine.exe version 0.12.2, build 9371605

$ docker -v

Docker version 17.06.0-ce, build 02c1d87

$ docker-compose -v

docker-compose version 1.14.0, build c7bdf9e3

  1. Create a Docker Machine to hold Docker containers for the application. The below command specifies the following configuration (note that your desired configuration may vary depending on your local machine):
  2. use thevirtualboxdriver
  3. allow the virtual disk to grow to up to 100GB
  4. allocate 8GB RAM
  5. allocate 2 CPU cores
  6. name the machinesquaad

$ docker-machine create -d virtualbox --virtualbox-disk-size "100000" --virtualbox-memory "8000" --virtualbox-cpu-count "2" squaad

  1. After creating the Docker Machine, check that it is there as expected.

$ docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS

squaad * virtualbox Running tcp://192.168.99.100:2376 v17.10.0-ce

  1. Set your Docker Machine environment tosquaad(or whatever you named your machine above - the Docker Toolbox default name isdefault). The setting only affects the current shell session - you must run it every time you open a new shell you wish to interact with thesquaadDocker Machine in.

$ eval $(docker-machine env squaad)

  1. Navigate to the top-level directory for the repo locally.

chris@DESKTOP-93IEIIB MINGW64 ~

$ cd ~/Documents/repos/SQUAAD/

chris@DESKTOP-93IEIIB MINGW64 ~/Documents/repos/SQUAAD (create-build)

$

  1. Run the command to bring up the containers and build the application.

$ docker-compose up --build

  1. Verify that the containers are up and running.

chris@DESKTOP-93IEIIB MINGW64 ~/Documents/repos/SQUAAD (create-build)

$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

d03e3c643c86 squaad_proxy "nginx -g 'daemon ..." 6 minutes ago Up 6 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp squaad_proxy_1

1bc8a85128e9 squaad_client "npm start" 6 minutes ago Up 6 minutes 3000/tcp squaad_client_1

78865526a98b squaad_server "/usr/entrypoint.s..." 4 hours ago Up 4 hours 8000/tcp squaad_server_1

ae38192dc02d postgres:9.6 "docker-entrypoint..." 22 hours ago Up 16 hours 5432/tcp squaad_postgres_1

  1. Navigate to the URL from step 3, in my case it is The 'https' is important here, and you can expect to see a warning about the self-signed cert locally (you will need to bypass it).

Linux & Mac Build

(tested on Xubuntu and Docker for Mac)

  1. Follow steps5,6, and7from 'Docker Toolbox - Windows' Instructions above.
  2. Navigate tolocalhost.

Auto Reload

Auto reload has been tested on the configurations above - code changes to the/srcdirectory in the server directory will trigger a reload of the API code. Similarly, code changes to the/srcdirectory in the client directory will reload the React. Things that will not auto reload -these will require a rebuild:

  • changes to package.json
  • adding/deleting entire files
  • changes to the .sql files in/db- these require thepostgres-datavolume to be removed to be recognized
  • changes toproxyconfiguration
  • Dockerfileanddocker-compose.ymlchanges

Generating Docs

Run the docs generation script (usesESDoc):

Client

chris@DESKTOP-93IEIIB MINGW64 ~/Documents/repos/squaad (code-docs)

$ docker-compose run client npm run esdoc

$ docker cp $(docker ps -aq | head -n 1):/tmp/esdoc/ ./client/docs

$ docker stop $(docker ps -aq | head -n 1)

$ docker rm $(docker ps -aq | head -n 1)

Server

chris@DESKTOP-93IEIIB MINGW64 ~/Documents/repos/squaad (code-docs)

$ docker-compose run server npm run esdoc

$ docker cp $(docker ps -aq | head -n 1):/tmp/esdoc/ ./server/docs

$ docker stop $(docker ps -aq | head -n 1)

$ docker rm $(docker ps -aq | head -n 1)

Docs are now available at./client/docs/esdocand./server/docs/esdoc.

Deploying on AWS

  1. SSH onto the server

ssh -i "squaad.pem"

  1. Elevate your shell privileges

sudo su

  1. Change directory to the SQUAAD repo

cd /home/ubuntu/squaad/SQUAAD

  1. Stash changes (only changes should be therealHTTP cert/key)

git stash

  1. Ensure you are on themasterbranch

git checkout master

  1. Pull the most recent changes

git pull

  1. Pop the stashed crt/key for HTTPS back (that way we don't commit this to the repo)

git stash pop

  1. Rebuild/redeploy the app with the newest code frommaster

HOST=squaadweb.com docker-compose up --build -d

Check Application Server Logs

  1. SSH onto the server

ssh -i "squaad.pem"

  1. Elevate your shell privileges

sudo su

  1. Change directory to the SQUAAD repo

cd /home/ubuntu/squaad/SQUAAD

  1. View logs

docker-compose logs -f

  1. Ctrl+cto exit

Clearing Out Docker Images to Regain Space (Optionally Run This After Deploy)

  1. SSH onto the server

ssh -i "squaad.pem"

  1. Elevate your shell privileges

sudo su

  1. Remove unused images

docker rmi $(docker images -a | grep "^<none>" | awk '{print $3}')

HTTPS Cert Generation

Overall Strategy

We use the Let's Encrypt certbot utility to generate a free and totally secure HTTPS cert. Certbotuses a series of tests, the easiest and most common being a placement of a specific file on anNGINX server via HTTP. Since our deployment redirects all HTTP requests to HTTPS by default, wetake the following approach:

-Ensure that squaadweb.com and are both pointing to the AWS server

-Spin down the application

-Spin up a generic NGINX Docker container, exposing port 80 to receive HTTP requests

-Install certbot within the NGINX container

-Run the cert generation utility

-Copy the certs out from the container onto the AWS server

-Put the certs into the proper placement for the application deployment

-Deploy the application as per usual

Requirements

-Root SSH access to the AWS server

-Familiarity with Docker (specifically docker cp)

Steps

ssh -i 'squaad.pem'

(on AWS server)

sudo su

docker stop $(docker ps -aq)

docker run -p 80:80 --name certbot_nginx -it nginx:1.13 bash

(in Docker container)

echo 'deb stretch-backports main' > etc/apt/sources.list

apt-get update

apt-get install python-certbot-nginx -t stretch-backports -y

certbot certonly --standalone -d squaadweb.com -d

(prompts for email and agreement to terms of service - answer as appropriate)

cp /etc/letsencrypt/live/squaadweb.com/fullchain.pem .

cp /etc/letsencrypt/live/squaadweb.com/privkey.pem .

<Ctrl+d>

(on AWS server)

docker cp certbot_nginx:fullchain.pem .

docker cp certbot_nginx:privkey.pem .

mkdir -p /etc/letsencrypt/live/squaadweb.com/

mv fullchain.pem /etc/letsencrypt/live/squaadweb.com

mv privkey.pem /etc/letsencrypt/live/squaadweb.com

cp /etc/letsencrypt/live/squaadweb.com/fullchain.pem /home/ubuntu/squaad/SQUAAD/proxy/ssl.crt

cp /etc/letsencrypt/live/squaadweb.com/privkey.pem /home/ubuntu/squaad/SQUAAD/proxy/ssl.key

cd /home/ubuntu/squaad/SQUAAD

HOST=squaadweb.com docker-compose up --build -d

docker rm certbot_nginx

(verify that both squaadweb.com and are shown as secure)

Clear Your Database and Rebuild

docker stop $(docker ps -aq)

docker rm -v squaad_postgres_1

docker volume rm squaad_postgres-data

docker-compose up --build

Note:You will lose your current local database changes and will be set back to the base structure with only the data in the migrations e.g. onlylogin and other default logins will be available.

The .sql scripts in this directory will only run on the very first time the database builds. Removing the data volume simulates the initial build and triggers the data load, which means our database version control is essentially a flush-and-fill currently.

Populating the DBwith New Records

  1. SCP the file onto the server, then SSH onto the server (for a server update)

scp -i "squaad.pem" huawei-request-insert :/home/ubuntu

ssh -i "squaad.pem"

  1. From a directory containing the dump file with only inserts (using huawei-request-insert as an example file)
  2. Determine the new projects to be insertedcat huawei-request-insert | grep "INSERT INTO applications";

example result:

INSERT INTO applications (repo_url, application) VALUES (' 'apache-carbondata');

INSERT INTO applications (repo_url, application) VALUES (' 'Huawei-openstack4j');

INSERT INTO applications (repo_url, application) VALUES (' 'hyperledger-fabric-sdk-java');

  1. Copy the new dump into the database container

docker cp huawei-request-insert squaad_postgres_1:/

  1. Drop pkey constraint on cpairs

docker exec squaad_postgres_1 psql -U postgres -c "

ALTER TABLE public.cpairs DROP CONSTRAINT cpairs_pkey;

"

  1. Drop pkey constraint on impact_pairs

docker exec squaad_postgres_1 psql -U postgres -c "

ALTER TABLE public.impact_pairs DROP CONSTRAINT impact_pairs_pkey;

"

  1. Run the new dump's SQL code to populate the tables

docker exec squaad_postgres_1 psql -U postgres -f huawei-request-insert

  1. Drop the impact_pairs and cpairs sequence index

docker exec squaad_postgres_1 psql -U postgres -c "

DROP SEQUENCE public.cpairs_id_seq CASCADE;

DROP SEQUENCE public.impact_pairs_id_seq CASCADE;

"

  1. Drop/add pkey constraint back on cpairs

docker exec squaad_postgres_1 psql -U postgres -c "

ALTER TABLE public.cpairs DROP COLUMN id;

ALTER TABLE public.cpairs ADD COLUMN id SERIAL PRIMARY KEY;

"

  1. Drop/add pkey constraint back on impact_pairs

docker exec squaad_postgres_1 psql -U postgres -c "

ALTER TABLE public.impact_pairs DROP COLUMN id;

ALTER TABLE public.impact_pairs ADD COLUMN id SERIAL PRIMARY KEY;

"

  1. Insert rows into the project table for each of the application datasets

docker exec squaad_postgres_1 psql -U postgres -c "

INSERT INTO project (name, repo_url, ispublic, prefix, compile_command, build_folder, source_folder, status) VALUES

('apache-carbondata', ' 1, '', '', '', '', 'ACTIVE');

INSERT INTO project (name, repo_url, ispublic, prefix, compile_command, build_folder, source_folder, status) VALUES

('Huawei-openstack4j', ' 1, '', '', '', '', 'ACTIVE');

INSERT INTO project (name, repo_url, ispublic, prefix, compile_command, build_folder, source_folder, status) VALUES

('hyperledger-fabric-sdk-java', ' 1, '', '', '', '', 'ACTIVE');

"

1

TM_ASBUILT_S18b_T01_V1.0Version Date: 04/27/2018