Skip to content

Commit e16c2f4

Browse files
刘志杰刘志杰
刘志杰
authored and
刘志杰
committed
remove dependency on DynamoDB, ElasticSearch and add Postgres as sole data source
1 parent d6ef493 commit e16c2f4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+13973
-2539
lines changed

‎.DS_Store

6 KB
Binary file not shown.

‎ReadMe.md

Lines changed: 55 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,10 @@ This microservice provides interaction with Challenge Resources.
1717

1818
## Related repos
1919
- [Challenge API](https://github.com/topcoder-platform/challenge-api)
20-
- [ES Processor](https://github.com/topcoder-platform/challenge-processor-es) - Updates data in ElasticSearch
21-
- [Legacy Processor](https://github.com/topcoder-platform/legacy-challenge-processor) - Moves data from DynamoDB back to Informix
22-
- [Legacy Migration Script](https://github.com/topcoder-platform/legacy-challenge-migration-script) - Moves data from Informix to DynamoDB
2320
- [Frontend App](https://github.com/topcoder-platform/challenge-engine-ui)
2421

2522
## Prerequisites
2623
- [NodeJS](https://nodejs.org/en/) (v10)
27-
- [DynamoDB](https://aws.amazon.com/dynamodb/)
28-
- [ElasticSearch](https://www.elastic.co/)
2924
- [Docker](https://www.docker.com/)
3025
- [Docker Compose](https://docs.docker.com/compose/)
3126

@@ -50,18 +45,8 @@ The following parameters can be set in config files or in env variables:
5045
- USER_API_URL: User api url, default is 'https://api.topcoder-dev.com/v3/users'
5146
- CHALLENGE_API_URL: Challenge api url, default is 'http://localhost:4000/v5/challenges'.
5247
- CHALLENGE_PHASES_API_URL: Challenge phases API URL, default is 'https://api.topcoder-dev.com/v5/challengephases'.
53-
- DYNAMODB.AWS_ACCESS_KEY_ID: The Amazon certificate key to use when connecting. Use local dynamodb you can set fake value
54-
- DYNAMODB.AWS_SECRET_ACCESS_KEY: The Amazon certificate access key to use when connecting. Use local dynamodb you can set fake value
55-
- DYNAMODB.AWS_REGION: The Amazon certificate region to use when connecting. Use local dynamodb you can set fake value
56-
- DYNAMODB.IS_LOCAL: Use Amazon DynamoDB Local or server.
57-
- DYNAMODB.URL: The local url if using Amazon DynamoDB Local
58-
- DYNAMODB.AWS_READ_UNITS: The DynamoDB table read unit configuration, default is 4
59-
- DYNAMODB.AWS_WRITE_UNITS: The DynamoDB table write unit configuration, default is 2
60-
- DYNAMODB.TIMEOUT: The timeout setting used in health check
48+
- SUBMISSIONS_API_URL: Submission API URL, default value is 'https://api.topcoder-dev.com/v5/submissions'
6149
- SCOPES: The M2M scopes, refer `config/default.js` for more information
62-
- OS.HOST: Opensearch host, default value is 'localhost:9200'
63-
- OS.OS_INDEX: Opensearch index name for resources, default value is 'resources'
64-
- OS.OS_REFRESH: Opensearch force refresh flag, default value is 'true'
6550
- BUSAPI_URL: the bus api, default value is 'https://api.topcoder-dev.com/v5'
6651
- KAFKA_ERROR_TOPIC: Kafka error topic, default value is 'common.error.reporting',
6752
- KAFKA_MESSAGE_ORIGINATOR: the Kafka message originator, default value is 'resources-api'
@@ -87,19 +72,22 @@ Configuration for testing is at `config/test.js`, only add such new configuratio
8772
*For the details of the supported format, please refer to https://www.npmjs.com/package/newman#reporters*.
8873

8974
## Available commands
75+
- Be sure to set correct value for environment variable `DATABASE_URL` first.
9076
- Install dependencies `npm install`
9177
- Run lint `npm run lint`
9278
- Run lint fix `npm run lint:fix`
9379
- Create tables `npm run create-tables`
94-
- Drop tables `npm run drop-tables`
80+
- Reset tables `npm run drop-tables`
81+
- Clear all data in db `npm run clear-tables`
9582
- Create tables for test environment `npm run create-tables:test`
96-
- Drop tables for test environment `npm run drop-tables:test`
83+
- Reset tables for test environment `npm run drop-tables:test`
9784
- Clear and init db `npm run init-db`
98-
- Initialize ElasticSearch `npm run init-es`
9985
- Start app `npm start`
10086
- App is running at `http://localhost:3000`
101-
- Start mock server `npm run mock-challenge-api`
102-
- The mock server is running at `http://localhost:4000`
87+
- Start mock challenge api server for unit tests `npm run mock-challenge-api`
88+
- Start mock api server for local dev `npm run mock-api`
89+
- The mock challenge api server is running at `http://localhost:4000`
90+
- The mock api server is running at `http://localhost:4001`
10391
- Run the Postman tests `npm run test:newman`
10492
- Clear the testing data by Postman tests: `npm run test:newman:clear`
10593

@@ -110,32 +98,43 @@ To install foreman follow this [link](https://theforeman.org/manuals/1.24/#3.Ins
11098
To know how to use foreman follow this [link](https://theforeman.org/manuals/1.24/#2.Quickstart)
11199

112100

113-
### DynamoDB Setup
101+
### Database Setup
114102

115-
We can use DynamoDB setup on Docker for testing purpose. Just run `docker-compose up` in `local` folder.
103+
We can use Postgres setup on Docker for testing purpose. Just run `docker-compose up` in `local` folder.
116104

117-
You can also use your own AWS DynamoDB service for testing purpose.
105+
You can also use docker to start it directly.
106+
```bash
107+
docker pull postgres:16.8
108+
109+
docker run -d --name resourcedb -p 5432:5432 \
110+
-e POSTGRES_USER=johndoe -e POSTGRES_DB=resourcedb \
111+
-e POSTGRES_PASSWORD=mypassword \
112+
postgres:16.8
113+
```
114+
115+
After that, please run
116+
```bash
117+
export DATABASE_URL="postgresql://johndoe:mypassword@localhost:5432/resourcedb?schema=public&statement_timeout=60000"
118+
```
118119

119120
### Create Tables
120121

121-
1. Make sure DynamoDB are running as per instructions above.
122+
1. Make sure Postgres are running as per instructions above.
122123
2. Make sure you have configured all config parameters. Refer [Configuration](#configuration)
123-
3. Run `npm run create-tables` to create tables.
124-
125-
### ElasticSearch Setup
124+
3. Run `npm run create-tables` to create tables and `npm run seed-tables` to create test data.
126125

127-
We can use ElasticSearch on Docker for testing purpose. Just run `docker-compose up` in `local` folder.
126+
### Mock API
128127

129-
You can also use your own remote ElasticSearch service for testing purpose.
128+
This mock service is designed for local development.
130129

131-
### Create ElasticSearch Index
130+
You can run mock api with `npm run mock-api`
132131

133-
1. Make sure ElasticSearch are running as per instructions above.
134-
2. Make sure you have configured all config parameters. Refer [Configuration](#configuration)
135-
3. Run `npm run init-es force` to create index.
132+
It will setup local environment for Challenge API, Submission API and Member API.
136133

137134
### Mock Challenge V5 API
138135

136+
This mock service is designed for unit tests.
137+
139138
The `GET /v5/challenges/{id}` is mocked. It is a simple server app, the code is under mock folder.
140139
You can start the mock server using command `npm run mock-challenge-api`.
141140

@@ -173,15 +172,32 @@ The following test parameters can be set in config file or in env variables:
173172

174173
### Prepare
175174

176-
- Start Local DynamoDB.
177-
- Create DynamoDB tables.
178-
- Start Local ElasticSearch.
179-
- Create ElasticSearch index.
175+
- Start Local Postgres.
176+
- Create Postgres tables and create test dat.
177+
- Config `DATABASE_URL`
180178
- Various config parameters should be properly set.
181179

182180
### Running unit tests
183181

184-
#### You need to `stop` the app server and mock API server before running unit tests.
182+
#### Setup Database for Tests
183+
184+
The Unit tests will clear all data in db. So it's better to setup db for test environment.
185+
186+
If you are using docker, you can run
187+
```bash
188+
docker run -d --name testdb -p 5430:5432 \
189+
-e POSTGRES_USER=johndoe -e POSTGRES_DB=testdb \
190+
-e POSTGRES_PASSWORD=mypassword \
191+
postgres:16.8
192+
193+
export DATABASE_URL="postgresql://johndoe:mypassword@localhost:5430/testdb?schema=public&statement_timeout=60000"
194+
```
195+
It will start a Postgres and listens to port `5430`. Technically you can run tests and application at the same time.
196+
197+
Be sure to export `DATABASE_URL` new value before running tests.
198+
199+
200+
#### Running unit tests.
185201

186202
To run unit tests and generate coverage report.
187203

@@ -207,7 +223,6 @@ npm run test
207223
# Then re-initialize the es server and the database.
208224

209225
$ npm run create-tables
210-
$ npm run init-es force
211226
$ npm run init-db
212227
```
213228

‎Verification.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,9 @@
22

33
## Postman test
44
- import Postman collection and environment in the docs folder to Postman
5-
- Refer `ReadMe.md` to start DynamoDB, create tables, start the app and mock server
6-
- run `npm run init-db` to clear the database before testing.
5+
- Refer `ReadMe.md` to start postgres, create tables, start the app and mock server
6+
- run `npm run init-db` to clear and setup the database before testing.
77
- Just run the whole test cases under provided environment.
8-
- For testing the negative scenario for health check endpoint, you can modify DYNAMODB.URL under `config/default.js` into an incorrect url or terminate the local DynamoDB. Then start the app again and run the health check test again.
98
- For each success POST/PUT/DELETE request, it will also send an event using bus api. You can also check the app console, it will have info log like `Publish event to Kafka topic <TOPIC_NAME>`. Go to https://lauscher.topcoder-dev.com/ view topics `challenge.action.resource.create`, `challenge.action.resource.delete`, `challenge.action.resource.role.create` and `challenge.action.resource.role.update`, to verify the Kafka message have successfully received.
109

1110
## Unit test Coverage

‎config/default.js

Lines changed: 0 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ module.exports = {
77
PORT: process.env.PORT || 3000,
88
API_VERSION: process.env.API_VERSION || 'v5',
99
DEFAULT_PAGE_SIZE: process.env.DEFAULT_PAGE_SIZE || 1000,
10-
MAX_ELASTIC_SEARCH_RECORDS_SIZE: process.env.MAX_ELASTIC_SEARCH_RECORDS_SIZE || 10000,
1110
// used to properly set the header response to api calls for services behind a load balancer
1211
API_BASE_URL: process.env.API_BASE_URL || `http://localhost:3000`,
1312

@@ -32,25 +31,6 @@ module.exports = {
3231
CHALLENGE_PHASES_API_URL: process.env.CHALLENGE_PHASES_API_URL || 'https://api.topcoder-dev.com/v5/challenge-phases',
3332
SUBMISSIONS_API_URL: process.env.SUBMISSIONS_API_URL || 'https://api.topcoder-dev.com/v5/submissions',
3433

35-
DYNAMODB: {
36-
AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID,
37-
AWS_SECRET_ACCESS_KEY: process.env.AWS_SECRET_ACCESS_KEY,
38-
AWS_REGION: process.env.AWS_REGION || 'us-east-1',
39-
IS_LOCAL_DB: process.env.IS_LOCAL_DB ? process.env.IS_LOCAL_DB === 'true' : true,
40-
DYNAMODB_URL: process.env.DYNAMODB_URL || 'http://localhost:7777',
41-
URL: process.env.DYNAMODB_URL || 'http://localhost:7777',
42-
AWS_READ_UNITS: process.env.AWS_READ_UNITS || 4,
43-
AWS_WRITE_UNITS: process.env.AWS_WRITE_UNITS || 2,
44-
TIMEOUT: process.env.DYNAMODB_TIMEOUT || 10000
45-
},
46-
47-
OS: {
48-
// above AWS_REGION is used if we use AWS OS
49-
HOST: process.env.OS_HOST || 'localhost:9200',
50-
OS_INDEX: process.env.OS_INDEX || 'resources',
51-
OS_REFRESH: process.env.OS_REFRESH || 'true'
52-
},
53-
5434
SCOPES: {
5535
READ: process.env.SCOPE_READ || 'read:resources',
5636
CREATE: process.env.SCOPE_CREATE || 'create:resources',

‎config/test.js

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,11 @@ module.exports = {
66
TERMS_API_URL: 'http://localhost:4000/v5/terms',
77
BUSAPI_URL: 'http://localhost:4000/v5',
88
CHALLENGE_PHASES_API_URL: 'http://localhost:4000/v5/challenge-phases',
9+
SUBMISSIONS_API_URL: 'http://localhost:4000/v5/submissions',
10+
MEMBER_API_URL: 'http://localhost:4000/v5/members',
911
WAIT_TIME: 6000,
1012
MOCK_CHALLENGE_API_PORT: 4000,
13+
MOCK_API_PORT: 4001,
1114
AUTH_V2_URL: process.env.AUTH_V2_URL || 'https://topcoder-dev.auth0.com/oauth/ro',
1215
AUTH_V2_CLIENT_ID: process.env.AUTH_V2_CLIENT_ID || '',
1316
AUTH_V3_URL: process.env.AUTH_V3_URL || 'https://api.topcoder-dev.com/v3/authorizations',

‎docs/dev.env

Lines changed: 1 addition & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,16 +18,7 @@
1818
CHALLENGE_API_URL: '',
1919
CHALLENGE_PHASES_API_URL: '',
2020

21-
DYNAMODB: {
22-
AWS_ACCESS_KEY_ID: '',
23-
AWS_SECRET_ACCESS_KEY: '',
24-
AWS_REGION: '',
25-
IS_LOCAL: '',
26-
URL: '',
27-
AWS_READ_UNITS: '',
28-
AWS_WRITE_UNITS: '',
29-
TIMEOUT: ''
30-
},
21+
DATABASE_URL: '',
3122

3223
SCOPES: {
3324
READ: '',

‎docs/prod.env

Lines changed: 1 addition & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,16 +18,7 @@
1818
CHALLENGE_API_URL: '',
1919
CHALLENGE_PHASES_API_URL: '',
2020

21-
DYNAMODB: {
22-
AWS_ACCESS_KEY_ID: '',
23-
AWS_SECRET_ACCESS_KEY: '',
24-
AWS_REGION: '',
25-
IS_LOCAL: '',
26-
URL: '',
27-
AWS_READ_UNITS: '',
28-
AWS_WRITE_UNITS: '',
29-
TIMEOUT: ''
30-
},
21+
DATABASE_URL: '',
3122

3223
SCOPES: {
3324
READ: '',

‎env.sh

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
#!/bin/sh
22

3-
export AUTH0_CLIENT_ID=
3+
export AUTH0_CLIENT_ID=jGIf2pd3f44B1jqvOai30BIKTZanYBfU
44

5-
export AUTH0_CLIENT_SECRET=
5+
export AUTH0_CLIENT_SECRET=ldzqVaVEbqhwjM5KtZ79sG8djZpAVK8Z7qieVcC3vRjI4NirgcinKSBpPwk6mYYP
66

77
export AUTH0_AUDIENCE=
88

@@ -24,6 +24,6 @@ export USER_CREDENTIALS_USERNAME=
2424

2525
export USER_CREDENTIALS_PASSWORD=
2626

27-
export IS_LOCAL_DB=
27+
export AUTH_SECRET=
2828

29-
export AUTH_SECRET=
29+
export DATABASE_URL=

‎local/docker-compose.yml

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,10 @@
11
version: '3'
22
services:
3-
dynamodb:
4-
image: tray/dynamodb-local
3+
postgres:
4+
image: postgres:16.8
55
ports:
6-
- "7777:7777"
7-
command: "-inMemory -port 7777"
8-
elasticsearch:
9-
image: elasticsearch:6.8.13
10-
ports:
11-
- "9200:9200"
12-
- "9300:9300"
6+
- "5432:5432"
137
environment:
14-
discovery.type: "single-node"
8+
POSTGRES_USER: johndoe
9+
POSTGRES_PASSWORD: mypassword
10+
POSTGRES_DB: resourcedb

0 commit comments

Comments
 (0)