diff --git a/README.md b/README.md index bf3a7be82..014a28b82 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,98 @@ NPM version NPM downloads +This is a fork of [node-postgres](https://github.com/brianc/node-postgres) which includes the following additional features: + +# Connection load balancing + +Users can use this feature in two configurations. + +### Cluster-aware / Uniform connection load balancing + +In the cluster-aware connection load balancing, connections are distributed across all the tservers in the cluster, irrespective of their placements. + +To enable the cluster-aware connection load balancing, provide the parameter `loadBalance` set to true or any as `loadBalance=any` in the connection url or the connection string (DSN style). [This section](#read-replica-cluster) explains the different values for `load_balance` parameter. + +``` +"postgresql://username:password@localhost:5433/database_name?loadBalance=any" +``` + +With this parameter specified in the url, the driver will fetch and maintain the list of tservers from the given endpoint (`localhost` in above example) available in the YugabyteDB cluster and distribute the connections equally across them. + +This list is refreshed every 5 minutes, when a new connection request is received. + +Application needs to use the same connection url to create every connection it needs, so that the distribution happens equally. + +### Topology-aware connection load balancing + +With topology-aware connnection load balancing, users can target tservers in specific zones by specifying these zones as `topologyKeys` with values in the format `cloudname.regionname.zonename`. Multiple zones can be specified as comma separated values. + +The connections will be distributed equally with the tservers in these zones. + +Note that, you would still need to specify `loadBalance` to one of the 5 allowed values to enable the topology-aware connection load balancing. + +``` +"postgresql://username:password@localhost:5433/database_name?loadBalance=true&topologyKeys=cloud1.region1.zone1,cloud1.region1.zone2" +``` +### Specifying fallback zones + +For topology-aware load balancing, you can now specify fallback placements too. This is not applicable for cluster-aware load balancing. +Each placement value can be suffixed with a colon (`:`) followed by a preference value between 1 and 10. +A preference value of `:1` means it is a primary placement. A preference value of `:2` means it is the first fallback placement and so on.If no preference value is provided, it is considered to be a primary placement (equivalent to one with preference value `:1`). Example given below. + +``` +"postgres://username:password@localhost:5433/database_name?loadBalance=true&topologyKeys=cloud1.region1.zone1:1,cloud1.region1.zone2:2"; +``` + +You can also use `*` for specifying all the zones in a given region as shown below. This is not allowed for cloud or region values. + +``` +"postgres://username:password@localhost:5433/database_name?loadBalance=true&topologyKeys=cloud1.region1.*:1,cloud1.region2.*:2"; +``` + +The driver attempts to connect to a node in following order: the least loaded node in the 1) primary placement(s), else in the 2) first fallback if specified, else in the 3) second fallback if specified and so on. +If no nodes are available either in primary placement(s) or in any of the fallback placements, then nodes in the rest of the cluster are attempted. +And this repeats for each connection request. + +## Specifying Refresh Interval + +Users can specify Refresh Time Interval, in seconds. It is the time interval between two attempts to refresh the information about cluster nodes. Default is 300. Valid values are integers between 0 and 600. Value 0 means refresh for each connection request. Any value outside this range is ignored and the default is used. + +To specify Refresh Interval, use the parameter `ybServersRefreshInterval` in the connection url or the connection string. + +``` +"postgres://username:password@localhost:5433/database_name?ybServersRefreshInterval=X&loadBalance=true&topologyKeys=cloud1.region1.*:1,cloud1.region2.*:2"; +``` +Here, X is the value of the refresh interval (seconds) in integer. + +## Other Connection Parameters: + +### fallback_to_topology_keys_only + +Applicable only for TopologyAware Load Balancing. When set to true, the smart driver does not attempt to connect to servers outside of primary and fallback placements specified via property. The default behaviour is to fallback to any available server in the entire cluster.(default value: false) + +### failed_host_reconnect_delay_secs + +The driver marks a server as failed with a timestamp, when it cannot connect to it. Later, whenever it refreshes the server list via yb_servers(), if it sees the failed server in the response, it marks the server as UP only if failed-host-reconnect-delay-secs time has elapsed. (The yb_servers() function does not remove a failed server immediately from its result and retains it for a while.)(default value: 5 seconds) + +## Read Replica Cluster + +node-postgres smart driver also enables load balancing across nodes in primary clusters which have associated Read Replica cluster. + +The connection property `loadBalance` allows five values using which users can distribute connections among different combination of nodes as per their requirements: + +- `only-rr` - Create connections only on Read Replica nodes +- `only-primary` - Create connections only on primary cluster nodes +- `prefer-rr` - Create connections on Read Replica nodes. If none available, on any node in the cluster including primary cluster nodes +- `prefer-primary` - Create connections on primary cluster nodes. If none available, on any node in the cluster including Read Replica nodes +- `any` or `true` - Equivalent to value true. Create connections on any node in the primary or Read Replica cluster + +default value is false + +To know more visit the [docs page](https://docs.yugabyte.com/preview/drivers-orms/). + +For a working example which demonstrates the configurations of connection load balancing see the [driver-examples](https://github.com/yugabyte/driver-examples/tree/main/nodejs) repository. + Non-blocking PostgreSQL client for Node.js. Pure JavaScript and optional native libpq bindings. ## Monorepo diff --git a/packages/pg-pool/README.md b/packages/pg-pool/README.md index b5f20bae9..f88a51de9 100644 --- a/packages/pg-pool/README.md +++ b/packages/pg-pool/README.md @@ -5,7 +5,7 @@ A connection pool for node-postgres ## install ```sh -npm i pg-pool pg +npm i @yugabytedb/pg-pool @yugabytedb/pg ``` ## use @@ -15,7 +15,7 @@ npm i pg-pool pg to use pg-pool you must first create an instance of a pool ```js -var Pool = require('pg-pool') +var Pool = require('@yugabytedb/pg-pool') // by default the pool uses the same // configuration as whatever `pg` version you have installed @@ -39,7 +39,7 @@ var pool2 = new Pool({ //you can supply a custom client constructor //if you want to use the native postgres client -var NativeClient = require('pg').native.Client +var NativeClient = require('@yugabytedb/pg').native.Client var nativePool = new Pool({ Client: NativeClient }) //you can even pool pg-native clients directly @@ -51,7 +51,7 @@ var pgNativePool = new Pool({ Client: PgNativeClient }) The Pool constructor does not support passing a Database URL as the parameter. To use pg-pool on heroku, for example, you need to parse the URL into a config object. Here is an example of how to parse a Database URL. ```js -const Pool = require('pg-pool'); +const Pool = require('@yugabytedb/pg-pool'); const url = require('url') const params = url.parse(process.env.DATABASE_URL); diff --git a/packages/pg-pool/index.js b/packages/pg-pool/index.js index 0d7314eb6..42d8bd83b 100644 --- a/packages/pg-pool/index.js +++ b/packages/pg-pool/index.js @@ -86,7 +86,7 @@ class Pool extends EventEmitter { this.options.allowExitOnIdle = this.options.allowExitOnIdle || false this.options.maxLifetimeSeconds = this.options.maxLifetimeSeconds || 0 this.log = this.options.log || function () {} - this.Client = this.options.Client || Client || require('pg').Client + this.Client = this.options.Client || Client || require('@yugabytedb/pg').Client this.Promise = this.options.Promise || global.Promise if (typeof this.options.idleTimeoutMillis === 'undefined') { diff --git a/packages/pg-pool/package.json b/packages/pg-pool/package.json index d89c12c5e..2665f6ec9 100644 --- a/packages/pg-pool/package.json +++ b/packages/pg-pool/package.json @@ -1,6 +1,6 @@ { - "name": "pg-pool", - "version": "3.5.1", + "name": "@yugabytedb/pg-pool", + "version": "3.5.1-yb-1", "description": "Connection pool for node-postgres", "main": "index.js", "directories": { @@ -11,7 +11,7 @@ }, "repository": { "type": "git", - "url": "git://github.com/brianc/node-postgres.git", + "url": "git://github.com/yugabyte/node-postgres.git", "directory": "packages/pg-pool" }, "keywords": [ @@ -23,9 +23,9 @@ "author": "Brian M. Carlson", "license": "MIT", "bugs": { - "url": "https://github.com/brianc/node-pg-pool/issues" + "url": "https://github.com/yugabyte/node-pg-pool/issues" }, - "homepage": "https://github.com/brianc/node-pg-pool#readme", + "homepage": "https://github.com/yugabyte/node-pg-pool#readme", "devDependencies": { "bluebird": "3.4.1", "co": "4.6.0", @@ -36,5 +36,8 @@ }, "peerDependencies": { "pg": ">=8.0" + }, + "publishConfig": { + "access": "public" } } diff --git a/packages/pg/README.md b/packages/pg/README.md index b3158b570..2aa87533c 100644 --- a/packages/pg/README.md +++ b/packages/pg/README.md @@ -4,12 +4,14 @@ NPM version NPM downloads +This is a fork of [node-postgres](https://github.com/brianc/node-postgres) which includes smart feature like Cluster Aware and Topology Aware load balancing. To know more visit the [docs page](https://docs.yugabyte.com/preview/drivers-orms/). + Non-blocking PostgreSQL client for Node.js. Pure JavaScript and optional native libpq bindings. ## Install ```sh -$ npm install pg +$ npm install @yugabytedb/pg ``` --- diff --git a/packages/pg/lib/client.js b/packages/pg/lib/client.js index 589aa9f84..0d134dbfc 100644 --- a/packages/pg/lib/client.js +++ b/packages/pg/lib/client.js @@ -11,17 +11,66 @@ var ConnectionParameters = require('./connection-parameters') var Query = require('./query') var defaults = require('./defaults') var Connection = require('./connection') +const dns = require('dns') +const { logger } = require('./logger') +const YB_SERVERS_QUERY = 'SELECT * FROM yb_servers()' +const DEFAULT_FAILED_HOST_TTL_SECONDS = 5 + +class ServerInfo { + constructor(hostName, port, placementInfo, public_ip, node_type) { + this.hostName = hostName + this.port = port + this.placementInfo = placementInfo + this.public_ip = public_ip + this.node_type = node_type + } +} + +class Lock { + constructor() { + this._locked = false + this._ee = new EventEmitter().setMaxListeners(0) + } + + acquire() { + return new Promise((resolve) => { + if (!this._locked) { + this._locked = true + return resolve() + } + const tryAcquire = () => { + if (!this._locked) { + this._locked = true + this._ee.removeListener('release', tryAcquire) + return resolve() + } + } + this._ee.on('release', tryAcquire) + }) + } + release() { + this._locked = false + setImmediate(() => this._ee.emit('release')) + } +} + +const lock = new Lock() class Client extends EventEmitter { constructor(config) { + logger.silly("Received connection string " + config) super() - this.connectionParameters = new ConnectionParameters(config) this.user = this.connectionParameters.user this.database = this.connectionParameters.database this.port = this.connectionParameters.port this.host = this.connectionParameters.host - + this.loadBalance = this.connectionParameters.loadBalance + this.topologyKeys = this.connectionParameters.topologyKeys + this.ybServersRefreshInterval = this.connectionParameters.ybServersRefreshInterval + this.fallbackToTopologyKeysOnly = this.connectionParameters.fallbackToTopologyKeysOnly + this.failedHostReconnectDelaySecs = this.connectionParameters.failedHostReconnectDelaySecs + this.connectionString = config // "hiding" the password so it doesn't show up in stack traces // or if the client is console.logged Object.defineProperty(this, 'password', { @@ -57,6 +106,11 @@ class Client extends EventEmitter { this.processID = null this.secretKey = null this.ssl = this.connectionParameters.ssl || false + this.config = config + // prevHostIfUsePublic will store the private host name before replacing + // it with public IP for making the connection + this.prevHostIfUsePublic = this.host + this.urlHost = this.host // As with Password, make SSL->Key (the private key) non-enumerable. // It won't show up in stack traces // or if the client is console.logged @@ -68,6 +122,25 @@ class Client extends EventEmitter { this._connectionTimeoutMillis = c.connectionTimeoutMillis || 0 } + // Control Connection + static controlClient = undefined + // Control Connection host + static controlClientHost = "" + static lastTimeMetaDataFetched = new Date().getTime() / 1000 + // Map of host -> connectionCount + static connectionMap = new Map() + // Map of failedHost -> ServerInfo of host + static failedHosts = new Map() + // Map of failedHost -> Time at which host was added to failedHosts Map + static failedHostsTime = new Map() + // Map of Primary Host -> ServerInfo + static hostServerInfoPrimary = new Map() + // Map of RR Host -> ServerInfo + static hostServerInfoRR = new Map() + // Boolean to check if public IP needs to be used or not + static usePublic = false + // Map of preference value as key and the list of placements as its value + static topologyKeyMap = new Map() _errorAllQueries(err) { const enqueueError = (query) => { @@ -85,11 +158,173 @@ class Client extends EventEmitter { this.queryQueue.length = 0 } + getLeastLoadedServer(hostsList) { + logger.silly("getLeastLoadedServer(): hostsList" + [...hostsList]) + if (hostsList.size === 0) { + return this.host + } + + let hostServerInfo; + + if (this.connectionParameters.loadBalance === 'any' || this.connectionParameters.loadBalance === 'true') { + hostServerInfo = new Map([...Client.hostServerInfoPrimary, ...Client.hostServerInfoRR]); + } else if (this.connectionParameters.loadBalance === 'only-primary' || this.connectionParameters.loadBalance === 'prefer-primary') { + hostServerInfo = new Map(Client.hostServerInfoPrimary); + } else { + hostServerInfo = new Map(Client.hostServerInfoRR); + } + logger.silly("Potential hosts: " + [...hostServerInfo]) + let minConnectionCount = Number.MAX_VALUE + let leastLoadedHosts = [] + for (var i = 1; i <= Client.topologyKeyMap.size; i++) { + let hosts = hostsList.keys() + for (let host of hosts) { + let placementInfoOfHost + if (hostServerInfo.has(host)) { + placementInfoOfHost = hostServerInfo.get(host).placementInfo + } else { + continue + } + var toCheckStar = placementInfoOfHost.split('.') + var starPlacementInfoOfHost = toCheckStar[0] + "." + toCheckStar[1] + ".*" + if (!Client.topologyKeyMap.get(i).includes(placementInfoOfHost) && !Client.topologyKeyMap.get(i).includes(starPlacementInfoOfHost)) { + continue + } + let hostCount + if (typeof hostsList.get(host) === 'object') { + hostCount = 0 + } else { + hostCount = hostsList.get(host) + } + if (minConnectionCount > hostCount) { + leastLoadedHosts = [] + minConnectionCount = hostCount + leastLoadedHosts.push(host) + } else if (minConnectionCount === hostCount) { + leastLoadedHosts.push(host) + } + } + if (leastLoadedHosts.length != 0) { + break + } + } + + if (leastLoadedHosts.length === 0) { + if (!(this.connectionParameters.loadBalance === 'prefer-primary' || this.connectionParameters.loadBalance === 'prefer-rr')) { + if (Client.topologyKeyMap.size === 0 || !this.connectionParameters.fallbackToTopologyKeysOnly) { + leastLoadedHosts = this.getHosts(hostsList, hostServerInfo) + } + } else { + leastLoadedHosts = this.getHosts(hostsList, hostServerInfo) + if (leastLoadedHosts.length === 0) { + if (this.connectionParameters.loadBalance === 'prefer-primary') { + leastLoadedHosts = this.getHosts(hostsList, new Map(Client.hostServerInfoRR)) + } else { + leastLoadedHosts = this.getHosts(hostsList, new Map(Client.hostServerInfoPrimary)) + } + } + } + } + + if (leastLoadedHosts.length === 0) { + throw new Error('Could not find a least loaded server.') + } + let randomIdx = Math.floor(Math.random() * leastLoadedHosts.length - 1) + 1 + let leastLoadedHost = leastLoadedHosts[randomIdx] + logger.silly("Least loaded servers are " + leastLoadedHosts) + logger.debug("Returning " + leastLoadedHost + " as the least loaded host") + return leastLoadedHost + } + + getHosts(hostsList, hostServerInfo) { + let minConnectionCount = Number.MAX_VALUE + let leastLoadedHosts = [] + + let hosts = hostsList.keys() + for (let value of hosts) { + if (!hostServerInfo.has(value)) { + continue + } + let hostCount + if (typeof hostsList.get(value) === 'object') { + hostCount = 0 + } else { + hostCount = hostsList.get(value) + } + if (minConnectionCount > hostCount) { + leastLoadedHosts = [] + minConnectionCount = hostCount + leastLoadedHosts.push(value) + } else if (minConnectionCount === hostCount) { + leastLoadedHosts.push(value) + } + } + return leastLoadedHosts + } + + isValidKey(key) { + var zones = key.split(':') + if (zones.length == 0 || zones.length > 2) { + logger.warn("Given topology-key " + key + " is invalid") + return false + } + var keyParts = zones[0].split('.') + if (keyParts.length !== 3) { + logger.warn("Given topology-key " + key + " is invalid") + return false + } + if (zones[1] == undefined) { + zones[1] = '1' + } + zones[1] = Number(zones[1]) + if (zones[1] < 1 || zones[1] > 10 || isNaN(zones[1]) || !Number.isInteger(zones[1])) { + logger.warn("Given topology-key " + key + " is invalid") + return false + } + logger.silly("Given topology-key " + key + " is valid") + return true + } + + incrementConnectionCount() { + let prevCount = 0 + let host = this.host + if (Client.usePublic) { + host = this.prevHostIfUsePublic + } + if (Client.connectionMap.has(host)) { + prevCount = Client.connectionMap.get(host) + } else if (Client.failedHosts.has(host)) { + logger.debug("Removing " + host + " from failed host list") + let serverInfo = Client.failedHosts.get(host) + if (serverInfo.node_type === 'primary') { + Client.hostServerInfoPrimary.set(host, serverInfo) + } else if (serverInfo.node_type === 'read_replica') { + Client.hostServerInfoRR.set(host, serverInfo) + } + Client.failedHosts.delete(host) + Client.failedHostsTime.delete(host) + } + Client.connectionMap.set(host, prevCount + 1) + logger.debug("Increasing connection count of " + host + " by 1") + } + _connect(callback) { + logger.silly("connect() is called") var self = this + if (this.connectionParameters.loadBalance !== 'false' && this._connecting) { + this.connection = + this.config.connection || + new Connection({ + stream: this.config.stream, + ssl: this.connectionParameters.ssl, + keepAlive: this.config.keepAlive || false, + keepAliveInitialDelayMillis: this.config.keepAliveInitialDelayMillis || 0, + encoding: this.connectionParameters.client_encoding || 'utf8', + }) + this._connecting = false + } var con = this.connection this._connectionCallback = callback - if (this._connecting || this._connected) { const err = new Error('Client has already been connected. You cannot reuse a client.') process.nextTick(() => { @@ -106,7 +341,34 @@ class Client extends EventEmitter { con.stream.destroy(new Error('timeout expired')) }, this._connectionTimeoutMillis) } - + if (this.connectionParameters.loadBalance !== 'false') { + if (Client.connectionMap.size && (Client.hostServerInfoPrimary.size || Client.hostServerInfoRR.size)) { + this.host = this.getLeastLoadedServer(Client.connectionMap) + if (Client.hostServerInfoPrimary.has(this.host)) { + this.port = Client.hostServerInfoPrimary.get(this.host).port + } else if (Client.hostServerInfoRR.has(this.host)) { + this.port = Client.hostServerInfoRR.get(this.host).port + } + logger.silly("Least loaded host received " + this.host + " port " + this.port) + } else if (Client.failedHosts.size) { + //ToDo: Why call getLeastLoadedServer with the failedHosts Map? Is this still required? + this.host = this.getLeastLoadedServer(Client.failedHosts) + this.port = Client.failedHosts.get(this.host).port + logger.silly("Least loaded host from failed host list received " + this.host + " port " + this.port) + } + } + if (Client.usePublic) { + let currentHost = this.host + let serverInfo + if (Client.hostServerInfoPrimary.has(this.host)) { + serverInfo = Client.hostServerInfoPrimary.get(currentHost) + } else if (Client.hostServerInfoRR.has(this.host)) { + serverInfo = Client.hostServerInfoRR.get(currentHost) + } + this.prevHostIfUsePublic = currentHost + this.host = serverInfo.public_ip + logger.silly("Using public ips, host " + this.host) + } if (this.host && this.host.indexOf('/') === 0) { con.connect(this.host + '/.s.PGSQL.' + this.port) } else { @@ -146,6 +408,10 @@ class Client extends EventEmitter { this._handleErrorEvent(error) } } else if (!this._connectionError) { + if (Client.controlClientHost === this.host && Client.controlClient != undefined) { + logger.silly("Control Connection host might be down, marking control connection as undefined") + Client.controlClient = undefined + } this._handleErrorEvent(error) } } @@ -156,23 +422,456 @@ class Client extends EventEmitter { }) } - connect(callback) { + attachErrorListenerOnClientConnection(client) { + client.on('error', () => { + if (Client.hostServerInfoPrimary.has(client.host)) { + logger.debug("Not able to connect to primary host " + client.host + ", adding it to failedHosts") + Client.failedHosts.set(client.host, Client.hostServerInfoPrimary.get(client.host)) + let start = new Date().getTime(); + Client.failedHostsTime.set(client.host, start) + Client.connectionMap.delete(client.host) + Client.hostServerInfoPrimary.delete(client.host) + } else if (Client.hostServerInfoRR.has(client.host)) { + logger.debug("Not able to connect to read replica host " + client.host + ", adding it to failedHosts") + Client.failedHosts.set(client.host, Client.hostServerInfoRR.get(client.host)) + let start = new Date().getTime(); + Client.failedHostsTime.set(client.host, start) + Client.connectionMap.delete(client.host) + Client.hostServerInfoRR.delete(client.host) + } + logger.silly("Control Connection host is down, marking control connection as undefined") + Client.controlClient = undefined + }) + } + + async iterateHostList(client) { + logger.silly("hostServerInfoPrimary: " + [...Client.hostServerInfoPrimary]) + logger.silly("hostServerInfoRR: " + [...Client.hostServerInfoRR]) + logger.silly("failedHosts: " + [...Client.failedHosts]) + let upHostsList = [...Client.hostServerInfoPrimary.keys(), ...Client.hostServerInfoRR.keys()][Symbol.iterator]() + let upHost = upHostsList.next() + let hostIsUp = false + while (upHost.value !== undefined && !hostIsUp) { + if (Client.failedHosts.has(upHost.value)) { + logger.silly(upHost.value + " is present in failed host list, trying next host") + upHost = upHostsList.next() + continue + } + client.host = upHost.value + client.connectionParameters.host = client.host + logger.debug("Trying to create control connection to " + client.host) + await client + .nowConnect() + .then((res) => { + hostIsUp = true + }) + .catch((err) => { + client.connection = + client.config.connection || + new Connection({ + stream: client.config.stream, + ssl: client.connectionParameters.ssl, + keepAlive: client.config.keepAlive || false, + keepAliveInitialDelayMillis: client.config.keepAliveInitialDelayMillis || 0, + encoding: client.connectionParameters.client_encoding || 'utf8', + }) + logger.debug("Not able to create control connection to host " + client.host + " adding it to failedHosts") + Client.failedHosts.set(client.host, Client.hostServerInfo.get(client.host)) + let start = new Date().getTime(); + Client.failedHostsTime.set(client.host, start) + Client.connectionMap.delete(client.host) + Client.hostServerInfoPrimary.delete(client.host) + Client.hostServerInfoRR.delete(client.host) + client._connecting = false + upHost = upHostsList.next() + }) + } + if (!hostIsUp) { + logger.debug("Not able to create control connection to any host in the cluster") + throw new Error('Not able to create control connection to any host in the cluster') + } + } + + async getConnection() { + logger.silly("Creating control connection...") + let currConnectionString = this.connectionString + let client; + if (typeof currConnectionString !== "string") { + currConnectionString.connectionTimeoutMillis = 10000 + client = new Client(currConnectionString) + } else { + client = new Client({ + connectionString: currConnectionString, + connectionTimeoutMillis: 10000, + }); + } + this.attachErrorListenerOnClientConnection(client) + let lookup = util.promisify(dns.lookup) + let addresses = [{ address: client.host }] + await lookup(client.host, { family: 0, all: true }).then((res) => { + addresses = res + }) + client.host = addresses[0].address // If both resolved then - IPv6 else IPv4 + client.loadBalance = 'false' + client.connectionParameters.loadBalance = 'false' + client.topologyKeys = '' + client.connectionParameters.topologyKeys = '' + if (Client.failedHosts.has(client.host)) { + await this.iterateHostList(client) + } else { + client.connectionParameters.host = client.host + logger.debug("Attempting to create control connection to " + client.host) + await client.nowConnect().catch(async (err) => { + client.connection = + client.config.connection || + new Connection({ + stream: client.config.stream, + ssl: client.connectionParameters.ssl, + keepAlive: client.config.keepAlive || false, + keepAliveInitialDelayMillis: client.config.keepAliveInitialDelayMillis || 0, + encoding: client.connectionParameters.client_encoding || 'utf8', + }) + client._connecting = false + logger.silly("Got error: " + err.message + " when attempting to create control connection to " + client.host) + if (addresses.length === 2) { + // If both resolved + client.host = addresses[1].address // IPv4 + if (Client.failedHosts.has(client.host)) { + await this.iterateHostList(client) + } else { + client.connectionParameters.host = client.host + logger.silly("Attempting to create control connection to " + client.host) + await client.nowConnect() + } + } + }) + } + Client.controlClientHost = client.host + logger.debug("Created control connection to host " + client.host) + return client + } + + async getServersInfo() { + logger.silly("Refreshing server info") + var client = Client.controlClient + var result + await client + .query({ + text: YB_SERVERS_QUERY, + statement_timeout: 10000, // Timeout after 10 seconds + }) + .then((res) => { + result = res + }) + .catch((err) => { + this.getConnection() + .then(async (res) => { + Client.controlClient = res + await this.getServersInfo() + }) + .catch((err) => { + return this.nowConnect(callback) + }) + }) + return result + } + + createServersList(data) { + logger.silly("Creating servers list") + Client.hostServerInfoPrimary.clear() + Client.hostServerInfoRR.clear() + data.forEach((eachServer) => { + var placementInfo = eachServer.cloud + '.' + eachServer.region + '.' + eachServer.zone + var nodeType = eachServer.node_type + var server = new ServerInfo(eachServer.host, eachServer.port, placementInfo, eachServer.public_ip, eachServer.node_type) + if (nodeType === 'primary') { + Client.hostServerInfoPrimary.set(eachServer.host, server) + if (eachServer.public_ip === this.host) { + Client.usePublic = true + } + } else { + Client.hostServerInfoRR.set(eachServer.host, server) + if (eachServer.public_ip === this.host) { + Client.usePublic = true + } + } + }) + logger.debug("Updated hostServerInfoPrimary to " + [...Client.hostServerInfoPrimary] + " and usePublic to " + Client.usePublic) + logger.debug("Updated hostServerInfoRR to " + [...Client.hostServerInfoRR] + " and usePublic to " + Client.usePublic) + } + + createConnectionMap(data) { + logger.silly("Creating connection map") + const currConnectionMap = new Map(Client.connectionMap) + Client.connectionMap.clear() + data.forEach((eachServer) => { + if (!Client.failedHosts.has(eachServer.host)) { + if (currConnectionMap.has(eachServer.host)) { + Client.connectionMap.set(eachServer.host, currConnectionMap.get(eachServer.host)) + } else { + Client.connectionMap.set(eachServer.host, 0) + } + } else { + let start = new Date().getTime(); + if (start - Client.failedHostsTime.get(eachServer.host) > (this.connectionParameters.failedHostReconnectDelaySecs * 1000)) { + logger.debug("Removing " + eachServer.host + " from failed host list") + Client.connectionMap.set(eachServer.host, 0) + Client.failedHosts.delete(eachServer.host) + Client.failedHostsTime.delete(eachServer.host) + } + } + }) + logger.debug("Updated connection map " + [...Client.connectionMap]) + } + + createTopologyKeyMap() { + logger.silly("Creating Topology key map") + var seperatedKeys = this.connectionParameters.topologyKeys.split(',') + for (let idx = 0; idx < seperatedKeys.length; idx++) { + let key = seperatedKeys[idx] + if (this.isValidKey(key)) { + var zones = key.split(':') + if (zones[1] == undefined) { + zones[1] = '1' + } + zones[1] = parseInt(zones[1]) + if (Client.topologyKeyMap.has(zones[1])) { + let currentzones = Client.topologyKeyMap.get(zones[1]) + currentzones.push(zones[0]) + Client.topologyKeyMap.set(zones[1], currentzones) + } else { + Client.topologyKeyMap.set(zones[1], [zones[0]]) + } + } else { + throw new Error('Bad Topology Key found - ' + key) + } + } + logger.debug("Updated topologyKey Map " + [...Client.topologyKeyMap]) + } + + createMetaData(data) { + logger.silly("Creating metadata ...") + this.createServersList(data) + Client.lastTimeMetaDataFetched = new Date().getTime() / 1000 + this.createConnectionMap(data) + if (this.connectionParameters.topologyKeys !== '') { + this.createTopologyKeyMap() + } + } + + nowConnect(callback) { + logger.silly("nowConnect() is called...") if (callback) { - this._connect(callback) - return + logger.silly("callback is not null") + if (this.connectionParameters.loadBalance !== 'false') { + this._connect((error) => { + if (error) { + if (this.connectionParameters.loadBalance !== 'false') { + if (Client.hostServerInfoPrimary.has(this.host)) { + logger.debug("Adding " + this.host + " to failed host list") + Client.failedHosts.set(this.host, Client.hostServerInfoPrimary.get(this.host)) + let start = new Date().getTime(); + Client.failedHostsTime.set(this.host, start) + Client.connectionMap.delete(this.host) + Client.hostServerInfoPrimary.delete(this.host) + } else if (Client.hostServerInfoRR.has(this.host)) { + logger.debug("Adding " + this.host + " to failed host list") + Client.failedHosts.set(this.host, Client.hostServerInfoRR.get(this.host)) + let start = new Date().getTime(); + Client.failedHostsTime.set(this.host, start) + Client.connectionMap.delete(this.host) + Client.hostServerInfoRR.delete(this.host) + } else if (Client.failedHosts.has(this.host)) { + logger.silly("Removing host " + this.host + " from failed hosts") + Client.failedHosts.delete(this.host) + Client.failedHostsTime.delete(this.host) + } + lock.release() + this.connect(callback) + } else { + callback(error) + return + } + } else { + if (this.connectionParameters.loadBalance !== 'false') { + lock.release() + this.incrementConnectionCount() + } + callback() + } + }) + return + } else { + this._connect(callback) + return + } } return new this._Promise((resolve, reject) => { this._connect((error) => { if (error) { - reject(error) + logger.silly("Not able to connect to " + this.host + " due to error " + error.message) + if (this.connectionParameters.loadBalance !== 'false' && (Client.hostServerInfoPrimary.size !== 0 || Client.hostServerInfoRR.size !== 0)) { + if (Client.hostServerInfoPrimary.has(this.host)) { + logger.debug("Adding " + this.host + " to failed host list") + Client.failedHosts.set(this.host, Client.hostServerInfoPrimary.get(this.host)) + let start = new Date().getTime(); + Client.failedHostsTime.set(this.host, start) + Client.connectionMap.delete(this.host) + Client.hostServerInfoPrimary.delete(this.host) + } else if (Client.hostServerInfoRR.has(this.host)) { + logger.debug("Adding " + this.host + " to failed host list") + Client.failedHosts.set(this.host, Client.hostServerInfoRR.get(this.host)) + let start = new Date().getTime(); + Client.failedHostsTime.set(this.host, start) + Client.connectionMap.delete(this.host) + Client.hostServerInfoRR.delete(this.host) + } else if (Client.failedHosts.has(this.host)) { + //ToDo: Why remove the host from failedHosts? Is this still required? + logger.silly("Removing host " + this.host + " from failed host list") + Client.failedHosts.delete(this.host) + Client.failedHostsTime.delete(this.host) + } + lock.release() + this.connect(callback) + } else { + reject(error) + } } else { + if (this.connectionParameters.loadBalance !== 'false') { + lock.release() + this.incrementConnectionCount() + } resolve() } }) }) } + updateConnectionMapAfterRefresh() { + logger.silly("Updating connection map after refresh") + let hostsInfoList = [...Client.hostServerInfoPrimary.keys(), ...Client.hostServerInfoRR.keys()] + for (let eachHost of hostsInfoList) { + if (!Client.connectionMap.has(eachHost)) { + if (!Client.failedHosts.has(eachHost)) { + Client.connectionMap.set(eachHost, 0) + } else { + let start = new Date().getTime(); + if (start - Client.failedHostsTime.get(eachHost) > (DEFAULT_FAILED_HOST_TTL_SECONDS * 1000)) { + logger.debug("Removing " + eachHost + " from failed host list") + Client.connectionMap.set(eachHost, 0) + Client.failedHosts.delete(eachHost) + Client.failedHostsTime.delete(eachHost) + } + } + } + } + let connectionMapHostList = Client.connectionMap.keys() + for (let eachHost of connectionMapHostList) { + if (!Client.hostServerInfoPrimary.has(eachHost) && !Client.hostServerInfoRR.has(eachHost)) { + Client.connectionMap.delete(eachHost) + } + } + logger.debug("Updated connection Map after refresh " + [...Client.connectionMap]); + logger.debug("Updated failed host list after refresh " + [...Client.failedHosts]); + } + + updateMetaData(data) { + logger.silly("Updating MetaData") + this.createServersList(data) + Client.lastTimeMetaDataFetched = new Date().getTime() / 1000 + this.updateConnectionMapAfterRefresh() + } + + isRefreshRequired() { + let currentTime = new Date().getTime() / 1000 + let diff = Math.floor(currentTime - Client.lastTimeMetaDataFetched) + if (diff >= this.connectionParameters.ybServersRefreshInterval) { + logger.silly("Refresh is required") + return true + } else { + logger.silly("Refresh is not required") + return false + } + } + + connect(callback) { + if (this.connectionParameters.loadBalance === 'false') { + logger.silly("Loadbalance is false, falling to upstream behaviour") + return this.nowConnect(callback) + } + /* + ToDo: We are holding the lock until the user connection gets created. + This looks like an overkill, why is this required? + */ + lock.acquire().then(() => { + logger.silly("loadBalance: " + this.connectionParameters.loadBalance) + logger.silly("topologyKeys: " + this.connectionParameters.topologyKeys) + logger.silly("ybServersRefreshInterval: " + this.connectionParameters.ybServersRefreshInterval) + logger.silly("fallbackToTopologyKeysOnly: " + this.connectionParameters.fallbackToTopologyKeysOnly) + logger.silly("failedHostReconnectDelaySecs: " + this.connectionParameters.failedHostReconnectDelaySecs) + if (Client.controlClient === undefined) { + this.getConnection() + .then(async (res) => { + Client.controlClient = res + this.getServersInfo() + .catch((err) => { + logger.silly("Not able to get servers info due to error " + err.message) + return this.nowConnect(callback) + }) + .then((res) => { + try { + this.createMetaData(res.rows) + return this.nowConnect(callback) + } catch (err) { + if (err.message.includes('Bad Topology Key found')) { + throw err + } else { + //ToDo: Why not throw the error? + logger.debug("Error caught: " + err.message) + } + } + }) + }) + .catch((err) => { + logger.silly("Not able to create control connection to any node due to error " + err.message) + return this.nowConnect(callback) + }) + } else { + if (this.isRefreshRequired()) { + this.getServersInfo() + .then((res) => { + this.updateMetaData(res.rows) + if (this.connectionParameters.topologyKeys !== '') { + this.createTopologyKeyMap() + } + return this.nowConnect(callback) + }) + .catch((err) => { + logger.silly("Not able to get servers info, error " + err.message) + return this.nowConnect(callback) + }) + } else { + let res = this.nowConnect(callback); + + if (res instanceof Promise) { + return res + .then(result => { + return result; + }) + .catch(error => { + logger.silly("Releasing lock.") + lock.release(); + throw error; + }); + } else { + return res; + } + } + } + }) + } + _attachListeners(con) { // password request handling con.on('authenticationCleartextPassword', this._handleAuthCleartextPassword.bind(this)) @@ -297,6 +996,13 @@ class Client extends EventEmitter { _handleErrorWhileConnecting(err) { if (this._connectionError) { // TODO(bmc): this is swallowing errors - we shouldn't do this + if (this.connectionParameters.loadBalance !== 'false' || Client.controlClient === undefined) { + if (this._connectionCallback) { + return this._connectionCallback(err) + } + this.emit('error', err) + return + } return } this._connectionError = true @@ -605,6 +1311,17 @@ class Client extends EventEmitter { this.connection.end() } + lock.acquire().then(() => { + if (this.connectionParameters.loadBalance !== 'false') { + let prevCount = Client.connectionMap.get(this.host) + if (prevCount > 0) { + logger.debug("Decreasing connection count (" + prevCount + ") of " + this.host + " by 1") + Client.connectionMap.set(this.host, prevCount - 1) + } + } + lock.release() + }) + if (cb) { this.connection.once('end', cb) } else { diff --git a/packages/pg/lib/connection-parameters.js b/packages/pg/lib/connection-parameters.js index 165e6d5d3..96d8b2863 100644 --- a/packages/pg/lib/connection-parameters.js +++ b/packages/pg/lib/connection-parameters.js @@ -58,7 +58,6 @@ class ConnectionParameters { this.user = val('user', config) this.database = val('database', config) - if (this.database === undefined) { this.database = this.user } @@ -94,7 +93,54 @@ class ConnectionParameters { enumerable: false, }) } - + this.loadBalance = val('loadBalance', config) + this.topologyKeys = val('topologyKeys', config) + this.ybServersRefreshInterval = val('ybServersRefreshInterval', config) + this.fallbackToTopologyKeysOnly = val('fallbackToTopologyKeysOnly', config) + this.failedHostReconnectDelaySecs = val('failedHostReconnectDelaySecs', config) + + if (typeof this.loadBalance === 'string') { + switch (this.loadBalance.toLowerCase()) { + case 'true': + case 'any': + case 'prefer-primary': + case 'prefer-rr': + case 'only-primary': + case 'only-rr': + case 'false': + break; + default: + throw new Error('Invalid loadBalance value: Valid values are only-rr, only-primary, prefer-rr, prefer-primary, any or true'); + } + } else if (typeof this.loadBalance === 'boolean') { + if (this.loadBalance) { + this.loadBalance = 'true' + } else { + this.loadBalance = 'false' + } + } + if (this.topologyKeys !== '') { + if (this.loadBalance === 'false') { + throw new Error(' You need to enable Load Balance feature to use Topology Aware! ') + } + } + this.ybServersRefreshInterval = Number(this.ybServersRefreshInterval) + if (isNaN(this.ybServersRefreshInterval) || !Number.isInteger(this.ybServersRefreshInterval)) { + throw new Error(' You need to Enter valid Refresh Interval ') + } + if (this.ybServersRefreshInterval < 0 || this.ybServersRefreshInterval > 600) { + this.ybServersRefreshInterval = 300 + } + if (typeof this.fallbackToTopologyKeysOnly === 'string') { + this.fallbackToTopologyKeysOnly = this.fallbackToTopologyKeysOnly === 'true' + } + this.failedHostReconnectDelaySecs = Number(this.failedHostReconnectDelaySecs) + if (isNaN(this.failedHostReconnectDelaySecs) || !Number.isInteger(this.failedHostReconnectDelaySecs)) { + throw new Error('Enter a valid value for failedHostReconnectDelaySecs') + } + if (this.failedHostReconnectDelaySecs < 0 || this.failedHostReconnectDelaySecs > 60) { + this.failedHostReconnectDelaySecs = 5 + } this.client_encoding = val('client_encoding', config) this.replication = val('replication', config) // a domain socket begins with '/' @@ -132,6 +178,11 @@ class ConnectionParameters { add(params, this, 'fallback_application_name') add(params, this, 'connect_timeout') add(params, this, 'options') + add(params, this, 'loadBalance') + add(params, this, 'topologyKeys') + add(params, this, 'ybServersRefreshInterval') + add(params, this, 'fallbackToTopologyKeysOnly') + add(params, this, 'failedHostReconnectDelaySecs') var ssl = typeof this.ssl === 'object' ? this.ssl : this.ssl ? { sslmode: this.ssl } : {} add(params, ssl, 'sslmode') diff --git a/packages/pg/lib/defaults.js b/packages/pg/lib/defaults.js index 9384e01cb..336839397 100644 --- a/packages/pg/lib/defaults.js +++ b/packages/pg/lib/defaults.js @@ -19,7 +19,22 @@ module.exports = { connectionString: undefined, // database port - port: 5432, + port: 5433, + + // Load Balance feature variable + loadBalance: 'false', + + // Topology keys + topologyKeys: '', + + // Refresh Interval + ybServersRefreshInterval: 300, + + // Fallback to Topology keys only + fallbackToTopologyKeysOnly: false, + + // Failed host reconnect delay seconds + failedHostReconnectDelaySecs: 5, // number of rows to return at a time from a prepared statement's // portal. 0 will return all rows at once diff --git a/packages/pg/lib/logger.js b/packages/pg/lib/logger.js new file mode 100644 index 000000000..9dcaca0b3 --- /dev/null +++ b/packages/pg/lib/logger.js @@ -0,0 +1,55 @@ +'use strict' + +const winston = require('winston'); + +/* + +Users can set the log level either by using an enviornment variable: + +`LOG_LEVEL = debug` + +or by using `setLogLevel` method:` + +const { logger, setLogLevel } = require('./logger'); + +// Change log level dynamically +setLogLevel('debug'); + +// Log messages +logger.debug('This is a debug message'); +logger.info('This is an info message'); + +*/ + +// Set the initial log level from the environment variable, default to 'info' +let logLevel = process.env.LOG_LEVEL || 'info'; + +// Configure the winston logger +var logger = winston.createLogger({ + level: logLevel, + levels: winston.config.npm.levels, + format: winston.format.combine( + winston.format.colorize(), + winston.format.timestamp(), + winston.format.printf(({ timestamp, level, message }) => { + return `${timestamp} [${level}]: ${message}`; + }) + ), + transports: [ + new winston.transports.Console(), + new winston.transports.File({ filename: 'app.log' }) + ] +}); + +// Function to set the log level dynamically and update the environment variable +var setLogLevel = (level) => { + logLevel = level; + process.env.LOG_LEVEL = level; // Update the environment variable + logger.level = level; // Update the logger's level + logger.info('Log level set to ' + level); +}; + +module.exports = { + logger, + setLogLevel +}; \ No newline at end of file diff --git a/packages/pg/package.json b/packages/pg/package.json index acc5e5f9a..31d75d6f0 100644 --- a/packages/pg/package.json +++ b/packages/pg/package.json @@ -1,7 +1,7 @@ { - "name": "pg", - "version": "8.7.3", - "description": "PostgreSQL client - pure javascript & libpq with the same API", + "name": "@yugabytedb/pg", + "version": "8.7.3-yb-9", + "description": "Pure JavaScript PostgreSQL client and native libpq bindings with YugabyteDB smart-driver features", "keywords": [ "database", "libpq", @@ -9,15 +9,20 @@ "postgre", "postgres", "postgresql", + "yugabytedb", "rdbms" ], - "homepage": "https://github.com/brianc/node-postgres", + "homepage": "https://github.com/yugabyte/node-postgres", "repository": { "type": "git", - "url": "git://github.com/brianc/node-postgres.git", + "url": "git://github.com/yugabyte/node-postgres.git", "directory": "packages/pg" }, "author": "Brian Carlson ", + "contributors": [ + "Priyanshi Gupta ", + "Harsh Daryani " + ], "main": "./lib", "dependencies": { "buffer-writer": "2.0.0", @@ -26,7 +31,8 @@ "pg-pool": "^3.5.1", "pg-protocol": "^1.5.0", "pg-types": "^2.1.0", - "pgpass": "1.x" + "pgpass": "1.x", + "winston": "^3.14.2" }, "devDependencies": { "async": "0.9.0", @@ -52,5 +58,8 @@ "license": "MIT", "engines": { "node": ">= 8.0.0" + }, + "publishConfig": { + "access": "public" } } diff --git a/yarn.lock b/yarn.lock index 3150a8804..e73d25c6b 100644 --- a/yarn.lock +++ b/yarn.lock @@ -23,6 +23,20 @@ chalk "^2.0.0" js-tokens "^4.0.0" +"@colors/colors@1.6.0", "@colors/colors@^1.6.0": + version "1.6.0" + resolved "https://registry.yarnpkg.com/@colors/colors/-/colors-1.6.0.tgz#ec6cd237440700bc23ca23087f513c75508958b0" + integrity sha512-Ir+AOibqzrIsL6ajt3Rz3LskB7OiMVHqltZmspbW/TJuTVuyOMirVqAkjfY6JISiLHgyNqicAC8AyHHGzNd/dA== + +"@dabh/diagnostics@^2.0.2": + version "2.0.3" + resolved "https://registry.yarnpkg.com/@dabh/diagnostics/-/diagnostics-2.0.3.tgz#7f7e97ee9a725dffc7808d93668cc984e1dc477a" + integrity sha512-hrlQOIi7hAfzsMqlGSFyVucrx38O+j6wiGOf//H2ecvIEqYN4ADBSS2iLMh5UFyDunCNniUIPk/q3riFv45xRA== + dependencies: + colorspace "1.1.x" + enabled "2.0.x" + kuler "^2.0.0" + "@eslint/eslintrc@^0.1.3": version "0.1.3" resolved "https://registry.yarnpkg.com/@eslint/eslintrc/-/eslintrc-0.1.3.tgz#7d1a2b2358552cc04834c0979bd4275362e37085" @@ -1012,6 +1026,11 @@ "@types/node" "*" "@types/pg-types" "*" +"@types/triple-beam@^1.3.2": + version "1.3.5" + resolved "https://registry.yarnpkg.com/@types/triple-beam/-/triple-beam-1.3.5.tgz#74fef9ffbaa198eb8b588be029f38b00299caa2c" + integrity sha512-6WaYesThRMCl19iryMYP7/x2OVgCtbIVflDGFpWnb9irXI3UjYE4AzmYuiUKY1AJstGijoY+MgUszMgRxIYTYw== + "@typescript-eslint/eslint-plugin@^4.4.0": version "4.4.0" resolved "https://registry.yarnpkg.com/@typescript-eslint/eslint-plugin/-/eslint-plugin-4.4.0.tgz#0321684dd2b902c89128405cf0385e9fe8561934" @@ -1363,6 +1382,11 @@ async@1.x: resolved "https://registry.yarnpkg.com/async/-/async-1.5.2.tgz#ec6a61ae56480c0c3cb241c95618e20892f9672a" integrity sha1-7GphrlZIDAw8skHJVhjiCJL5Zyo= +async@^3.2.3: + version "3.2.6" + resolved "https://registry.yarnpkg.com/async/-/async-3.2.6.tgz#1b0728e14929d51b85b449b7f06e27c1145e38ce" + integrity sha512-htCUDlxyyCLMgaM3xXg0C0LW2xqfuQ6p05pCEIsXuyQ+a1koYKTuBMzRNwmybfLgvJDMd0r1LTn4+E0Ti6C2AA== + asynckit@^0.4.0: version "0.4.0" resolved "https://registry.yarnpkg.com/asynckit/-/asynckit-0.4.0.tgz#c79ed97f7f34cb8f2ba1bc9790bcc366474b4b79" @@ -1760,7 +1784,7 @@ collection-visit@^1.0.0: map-visit "^1.0.0" object-visit "^1.0.0" -color-convert@^1.9.0: +color-convert@^1.9.0, color-convert@^1.9.3: version "1.9.3" resolved "https://registry.yarnpkg.com/color-convert/-/color-convert-1.9.3.tgz#bb71850690e1f136567de629d2d5471deda4c1e8" integrity sha512-QfAUtd+vFdAtFQcC8CCyYt1fYWxSqAiK2cSD6zDB8N3cpsEBAvRxp9zOGg6G/SHHJYAT88/az/IuDGALsNVbGg== @@ -1779,11 +1803,35 @@ color-name@1.1.3: resolved "https://registry.yarnpkg.com/color-name/-/color-name-1.1.3.tgz#a7d0558bd89c42f795dd42328f740831ca53bc25" integrity sha1-p9BVi9icQveV3UIyj3QIMcpTvCU= -color-name@~1.1.4: +color-name@^1.0.0, color-name@~1.1.4: version "1.1.4" resolved "https://registry.yarnpkg.com/color-name/-/color-name-1.1.4.tgz#c2a09a87acbde69543de6f63fa3995c826c536a2" integrity sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA== +color-string@^1.6.0: + version "1.9.1" + resolved "https://registry.yarnpkg.com/color-string/-/color-string-1.9.1.tgz#4467f9146f036f855b764dfb5bf8582bf342c7a4" + integrity sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg== + dependencies: + color-name "^1.0.0" + simple-swizzle "^0.2.2" + +color@^3.1.3: + version "3.2.1" + resolved "https://registry.yarnpkg.com/color/-/color-3.2.1.tgz#3544dc198caf4490c3ecc9a790b54fe9ff45e164" + integrity sha512-aBl7dZI9ENN6fUGC7mWpMTPNHmWUSNan9tuWN6ahh5ZLNk9baLJOnSMlrQkHcrfFgz2/RigjUVAjdx36VcemKA== + dependencies: + color-convert "^1.9.3" + color-string "^1.6.0" + +colorspace@1.1.x: + version "1.1.4" + resolved "https://registry.yarnpkg.com/colorspace/-/colorspace-1.1.4.tgz#8d442d1186152f60453bf8070cd66eb364e59243" + integrity sha512-BgvKJiuVu1igBUF2kEjRCZXol6wiiGbY5ipL/oVPwm0BL9sIpMIzM8IK7vwuxIIzOXMV3Ey5w+vxhm0rR/TN8w== + dependencies: + color "^3.1.3" + text-hex "1.0.x" + columnify@^1.5.4: version "1.5.4" resolved "https://registry.yarnpkg.com/columnify/-/columnify-1.5.4.tgz#4737ddf1c7b69a8a7c340570782e947eec8e78bb" @@ -2244,6 +2292,11 @@ emoji-regex@^7.0.1: resolved "https://registry.yarnpkg.com/emoji-regex/-/emoji-regex-7.0.3.tgz#933a04052860c85e83c122479c4748a8e4c72156" integrity sha512-CwBLREIQ7LvYFB0WyRvwhq5N5qPhc6PMjD6bYggFlI5YyDgl+0vxq5VHbMOFqLg7hfWzmu8T5Z1QofhmTIhItA== +enabled@2.0.x: + version "2.0.0" + resolved "https://registry.yarnpkg.com/enabled/-/enabled-2.0.0.tgz#f9dd92ec2d6f4bbc0d5d1e64e21d61cd4665e7c2" + integrity sha512-AKrN98kuwOzMIdAizXGI86UFBoo26CL21UM763y1h/GMSJ4/OHU9k2YlsmBpyScFo/wbLzWQJBMCW4+IO3/+OQ== + encoding@^0.1.11: version "0.1.13" resolved "https://registry.yarnpkg.com/encoding/-/encoding-0.1.13.tgz#56574afdd791f54a8e9b2785c0582a2d26210fa9" @@ -2660,6 +2713,11 @@ fastq@^1.6.0: dependencies: reusify "^1.0.4" +fecha@^4.2.0: + version "4.2.3" + resolved "https://registry.yarnpkg.com/fecha/-/fecha-4.2.3.tgz#4d9ccdbc61e8629b259fdca67e65891448d569fd" + integrity sha512-OP2IUU6HeYKJi3i0z4A19kHMQoLVs4Hc+DPqqxI2h/DPZHTm/vjsfC6P0b4jCMy14XizLBqvndQ+UilD7707Jw== + figgy-pudding@^3.4.1, figgy-pudding@^3.5.1: version "3.5.2" resolved "https://registry.yarnpkg.com/figgy-pudding/-/figgy-pudding-3.5.2.tgz#b4eee8148abb01dcf1d1ac34367d59e12fa61d6e" @@ -2755,6 +2813,11 @@ flush-write-stream@^1.0.0: inherits "^2.0.3" readable-stream "^2.3.6" +fn.name@1.x.x: + version "1.1.0" + resolved "https://registry.yarnpkg.com/fn.name/-/fn.name-1.1.0.tgz#26cad8017967aea8731bc42961d04a3d5988accc" + integrity sha512-GRnmB5gPyJpAhTQdSZTSp9uaPSvl09KoYcMQtsB9rQoOmzs9dH6ffeccH+Z+cv6P68Hu5bC6JjRh4Ah/mHSNRw== + "fomatto@git://github.com/BonsaiDen/Fomatto.git#468666f600b46f9067e3da7200fd9df428923ea6": version "0.6.0" resolved "git://github.com/BonsaiDen/Fomatto.git#468666f600b46f9067e3da7200fd9df428923ea6" @@ -3367,6 +3430,11 @@ is-arrayish@^0.2.1: resolved "https://registry.yarnpkg.com/is-arrayish/-/is-arrayish-0.2.1.tgz#77c99840527aa8ecb1a8ba697b80645a7a926a9d" integrity sha1-d8mYQFJ6qOyxqLppe4BkWnqSap0= +is-arrayish@^0.3.1: + version "0.3.2" + resolved "https://registry.yarnpkg.com/is-arrayish/-/is-arrayish-0.3.2.tgz#4574a2ae56f7ab206896fb431eaeed066fdf8f03" + integrity sha512-eVRqCvVlZbuw3GrM63ovNSNAeA1K16kaR/LRY/92w0zxQ5/1YzwblUX652i4Xs9RwAGjW9d9y6X88t8OaAJfWQ== + is-binary-path@~2.1.0: version "2.1.0" resolved "https://registry.yarnpkg.com/is-binary-path/-/is-binary-path-2.1.0.tgz#ea1f7f3b80f064236e83470f86c09c254fb45b09" @@ -3549,6 +3617,11 @@ is-stream@^1.1.0: resolved "https://registry.yarnpkg.com/is-stream/-/is-stream-1.1.0.tgz#12d4a3dd4e68e0b79ceb8dbc84173ae80d91ca44" integrity sha1-EtSj3U5o4Lec6428hBc66A2RykQ= +is-stream@^2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/is-stream/-/is-stream-2.0.1.tgz#fac1e3d53b97ad5a9d0ae9cef2389f5810a5c077" + integrity sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg== + is-symbol@^1.0.2: version "1.0.3" resolved "https://registry.yarnpkg.com/is-symbol/-/is-symbol-1.0.3.tgz#38e1014b9e6329be0de9d24a414fd7441ec61937" @@ -3732,6 +3805,11 @@ kind-of@^6.0.0, kind-of@^6.0.2, kind-of@^6.0.3: resolved "https://registry.yarnpkg.com/kind-of/-/kind-of-6.0.3.tgz#07c05034a6c349fa06e24fa35aa76db4580ce4dd" integrity sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw== +kuler@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/kuler/-/kuler-2.0.0.tgz#e2c570a3800388fb44407e851531c1d670b061b3" + integrity sha512-Xq9nH7KlWZmXAtodXDDRE7vs6DU1gTU8zYDHDiWLSip45Egwq3plLHzPn27NgvzL2r1LMPC1vdqh98sQxtqj4A== + lcov-parse@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/lcov-parse/-/lcov-parse-1.0.0.tgz#eb0d46b54111ebc561acb4c408ef9363bdc8f7e0" @@ -3904,6 +3982,18 @@ log-symbols@3.0.0: dependencies: chalk "^2.4.2" +logform@^2.6.0, logform@^2.6.1: + version "2.6.1" + resolved "https://registry.yarnpkg.com/logform/-/logform-2.6.1.tgz#71403a7d8cae04b2b734147963236205db9b3df0" + integrity sha512-CdaO738xRapbKIMVn2m4F6KTj4j7ooJ8POVnebSgKo3KBz5axNXRAL7ZdRjIV6NOr2Uf4vjtRkxrFETOioCqSA== + dependencies: + "@colors/colors" "1.6.0" + "@types/triple-beam" "^1.3.2" + fecha "^4.2.0" + ms "^2.1.1" + safe-stable-stringify "^2.3.1" + triple-beam "^1.3.0" + loud-rejection@^1.0.0: version "1.6.0" resolved "https://registry.yarnpkg.com/loud-rejection/-/loud-rejection-1.6.0.tgz#5b46f80147edee578870f086d04821cf998e551f" @@ -4540,6 +4630,13 @@ once@1.x, once@^1.3.0, once@^1.3.1, once@^1.4.0: dependencies: wrappy "1" +one-time@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/one-time/-/one-time-1.0.0.tgz#e06bc174aed214ed58edede573b433bbf827cb45" + integrity sha512-5DXOiRKwuSEcQ/l0kGCF6Q3jcADFv5tSmRaJck/OqkVFcOzutB134KRSfF0xDrL39MNnqxbHBbUUcjZIhTgb2g== + dependencies: + fn.name "1.x.x" + onetime@^2.0.0: version "2.0.1" resolved "https://registry.yarnpkg.com/onetime/-/onetime-2.0.1.tgz#067428230fd67443b2794b22bba528b6867962d4" @@ -4830,6 +4927,16 @@ performance-now@^2.1.0: resolved "https://registry.yarnpkg.com/performance-now/-/performance-now-2.1.0.tgz#6309f4e0e5fa913ec1c69307ae364b4b377c9e7b" integrity sha1-Ywn04OX6kT7BxpMHrjZLSzd8nns= +pg-cloudflare@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/pg-cloudflare/-/pg-cloudflare-1.1.1.tgz#e6d5833015b170e23ae819e8c5d7eaedb472ca98" + integrity sha512-xWPagP/4B6BgFO+EKz3JONXv3YDgvkbVrGw2mTo3D6tVDQRh1e7cqVGvyR3BE+eQgAvx1XhW/iEASj4/jCWl3Q== + +pg-connection-string@^2.6.4: + version "2.6.4" + resolved "https://registry.yarnpkg.com/pg-connection-string/-/pg-connection-string-2.6.4.tgz#f543862adfa49fa4e14bc8a8892d2a84d754246d" + integrity sha512-v+Z7W/0EO707aNMaAEfiGnGL9sxxumwLl2fJvCQtMn9Fxsg+lPpPkdcyBSv/KFgpGdYkMfn+EI1Or2EHjpgLCA== + pg-copy-streams@0.3.0: version "0.3.0" resolved "https://registry.yarnpkg.com/pg-copy-streams/-/pg-copy-streams-0.3.0.tgz#a4fbc2a3b788d4e9da6f77ceb35422d8d7043b7f" @@ -4845,6 +4952,16 @@ pg-int8@1.0.1: resolved "https://registry.yarnpkg.com/pg-int8/-/pg-int8-1.0.1.tgz#943bd463bf5b71b4170115f80f8efc9a0c0eb78c" integrity sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw== +pg-pool@^3.5.1, pg-pool@^3.6.2: + version "3.6.2" + resolved "https://registry.yarnpkg.com/pg-pool/-/pg-pool-3.6.2.tgz#3a592370b8ae3f02a7c8130d245bc02fa2c5f3f2" + integrity sha512-Htjbg8BlwXqSBQ9V8Vjtc+vzf/6fVUuak/3/XXKA9oxZprwW3IMDQTGHP+KDmVL7rtd+R1QjbnCFPuTHm3G4hg== + +pg-protocol@^1.6.1: + version "1.6.1" + resolved "https://registry.yarnpkg.com/pg-protocol/-/pg-protocol-1.6.1.tgz#21333e6d83b01faaebfe7a33a7ad6bfd9ed38cb3" + integrity sha512-jPIlvgoD63hrEuihvIg+tJhoGjUsLPn6poJY9N5CnlPd91c2T18T/9zBtLxZSb1EhYxBRoZJtzScCaWlYLtktg== + pg-types@^2.1.0: version "2.2.0" resolved "https://registry.yarnpkg.com/pg-types/-/pg-types-2.2.0.tgz#2d0250d636454f7cfa3b6ae0382fdfa8063254a3" @@ -4856,6 +4973,19 @@ pg-types@^2.1.0: postgres-date "~1.0.4" postgres-interval "^1.1.0" +pg@^8.7.3: + version "8.12.0" + resolved "https://registry.yarnpkg.com/pg/-/pg-8.12.0.tgz#9341724db571022490b657908f65aee8db91df79" + integrity sha512-A+LHUSnwnxrnL/tZ+OLfqR1SxLN3c/pgDztZ47Rpbsd4jUytsTtwQo/TLPRzPJMp/1pbhYVhH9cuSZLAajNfjQ== + dependencies: + pg-connection-string "^2.6.4" + pg-pool "^3.6.2" + pg-protocol "^1.6.1" + pg-types "^2.1.0" + pgpass "1.x" + optionalDependencies: + pg-cloudflare "^1.1.1" + pgpass@1.x: version "1.0.2" resolved "https://registry.yarnpkg.com/pgpass/-/pgpass-1.0.2.tgz#2a7bb41b6065b67907e91da1b07c1847c877b306" @@ -5161,6 +5291,15 @@ read@1, read@~1.0.1: string_decoder "^1.1.1" util-deprecate "^1.0.1" +readable-stream@^3.4.0, readable-stream@^3.6.2: + version "3.6.2" + resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-3.6.2.tgz#56a9b36ea965c00c5a93ef31eb111a0f11056967" + integrity sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA== + dependencies: + inherits "^2.0.3" + string_decoder "^1.1.1" + util-deprecate "^1.0.1" + readdir-scoped-modules@^1.0.0: version "1.1.0" resolved "https://registry.yarnpkg.com/readdir-scoped-modules/-/readdir-scoped-modules-1.1.0.tgz#8d45407b4f870a0dcaebc0e28670d18e74514309" @@ -5387,6 +5526,11 @@ safe-regex@^1.1.0: dependencies: ret "~0.1.10" +safe-stable-stringify@^2.3.1: + version "2.5.0" + resolved "https://registry.yarnpkg.com/safe-stable-stringify/-/safe-stable-stringify-2.5.0.tgz#4ca2f8e385f2831c432a719b108a3bf7af42a1dd" + integrity sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA== + "safer-buffer@>= 2.1.2 < 3", "safer-buffer@>= 2.1.2 < 3.0.0", safer-buffer@^2.0.2, safer-buffer@^2.1.0, safer-buffer@~2.1.0: version "2.1.2" resolved "https://registry.yarnpkg.com/safer-buffer/-/safer-buffer-2.1.2.tgz#44fa161b0187b9549dd84bb91802f9bd8385cd6a" @@ -5458,6 +5602,13 @@ signal-exit@^3.0.0, signal-exit@^3.0.2: resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-3.0.3.tgz#a1410c2edd8f077b08b4e253c8eacfcaf057461c" integrity sha512-VUJ49FC8U1OxwZLxIbTTrDvLnf/6TDgxZcK8wxR8zs13xpx7xbG60ndBlhNrFi2EMuFRoeDoJO7wthSLq42EjA== +simple-swizzle@^0.2.2: + version "0.2.2" + resolved "https://registry.yarnpkg.com/simple-swizzle/-/simple-swizzle-0.2.2.tgz#a4da6b635ffcccca33f70d17cb92592de95e557a" + integrity sha512-JA//kQgZtbuY83m+xT+tXJkmJncGMTFT+C+g2h2R9uxkYIrE2yy9sgmcLhCnw57/WSD+Eh3J97FPEDFnbXnDUg== + dependencies: + is-arrayish "^0.3.1" + slash@^2.0.0: version "2.0.0" resolved "https://registry.yarnpkg.com/slash/-/slash-2.0.0.tgz#de552851a1759df3a8f206535442f5ec4ddeab44" @@ -5655,6 +5806,11 @@ ssri@^6.0.0, ssri@^6.0.1: dependencies: figgy-pudding "^3.5.1" +stack-trace@0.0.x: + version "0.0.10" + resolved "https://registry.yarnpkg.com/stack-trace/-/stack-trace-0.0.10.tgz#547c70b347e8d32b4e108ea1a2a159e5fdde19c0" + integrity sha512-KGzahc7puUKkzyMt+IqAep+TVNbKP+k2Lmwhub39m1AsTSkaDutx56aDCo+HLDzf/D26BIHTJWNiTG1KAJiQCg== + static-extend@^0.1.1: version "0.1.2" resolved "https://registry.yarnpkg.com/static-extend/-/static-extend-0.1.2.tgz#60809c39cbff55337226fd5e0b520f341f1fb5c6" @@ -5904,6 +6060,11 @@ text-extensions@^1.0.0: resolved "https://registry.yarnpkg.com/text-extensions/-/text-extensions-1.9.0.tgz#1853e45fee39c945ce6f6c36b2d659b5aabc2a26" integrity sha512-wiBrwC1EhBelW12Zy26JeOUkQ5mRu+5o8rpsJk5+2t+Y5vE7e842qtZDQ2g1NpX/29HdyFeJ4nSIhI47ENSxlQ== +text-hex@1.0.x: + version "1.0.0" + resolved "https://registry.yarnpkg.com/text-hex/-/text-hex-1.0.0.tgz#69dc9c1b17446ee79a92bf5b884bb4b9127506f5" + integrity sha512-uuVGNWzgJ4yhRaNSiubPY7OjISw4sw4E5Uv0wbjp+OzcbmVU/rsT8ujgcXJhn9ypzsgr5vlzpPqP+MBBKcGvbg== + text-table@^0.2.0: version "0.2.0" resolved "https://registry.yarnpkg.com/text-table/-/text-table-0.2.0.tgz#7f5ee823ae805207c00af2df4a84ec3fcfa570b4" @@ -6047,6 +6208,11 @@ trim-off-newlines@^1.0.0: resolved "https://registry.yarnpkg.com/trim-off-newlines/-/trim-off-newlines-1.0.3.tgz#8df24847fcb821b0ab27d58ab6efec9f2fe961a1" integrity sha512-kh6Tu6GbeSNMGfrrZh6Bb/4ZEHV1QlB4xNDBeog8Y9/QwFlKTRyWvY3Fs9tRDAMZliVUwieMgEdIeL/FtqjkJg== +triple-beam@^1.3.0: + version "1.4.1" + resolved "https://registry.yarnpkg.com/triple-beam/-/triple-beam-1.4.1.tgz#6fde70271dc6e5d73ca0c3b24e2d92afb7441984" + integrity sha512-aZbgViZrg1QNcG+LULa7nhZpJTZSLm/mXnHXnbAbjmN5aSa0y7V+wvv6+4WaBtpISJzThKy+PIPxc1Nq1EJ9mg== + ts-node@^8.5.4: version "8.10.2" resolved "https://registry.yarnpkg.com/ts-node/-/ts-node-8.10.2.tgz#eee03764633b1234ddd37f8db9ec10b75ec7fb8d" @@ -6330,6 +6496,32 @@ windows-release@^3.1.0: dependencies: execa "^1.0.0" +winston-transport@^4.7.0: + version "4.7.1" + resolved "https://registry.yarnpkg.com/winston-transport/-/winston-transport-4.7.1.tgz#52ff1bcfe452ad89991a0aaff9c3b18e7f392569" + integrity sha512-wQCXXVgfv/wUPOfb2x0ruxzwkcZfxcktz6JIMUaPLmcNhO4bZTwA/WtDWK74xV3F2dKu8YadrFv0qhwYjVEwhA== + dependencies: + logform "^2.6.1" + readable-stream "^3.6.2" + triple-beam "^1.3.0" + +winston@^3.14.2: + version "3.14.2" + resolved "https://registry.yarnpkg.com/winston/-/winston-3.14.2.tgz#94ce5fd26d374f563c969d12f0cd9c641065adab" + integrity sha512-CO8cdpBB2yqzEf8v895L+GNKYJiEq8eKlHU38af3snQBQ+sdAIUepjMSguOIJC7ICbzm0ZI+Af2If4vIJrtmOg== + dependencies: + "@colors/colors" "^1.6.0" + "@dabh/diagnostics" "^2.0.2" + async "^3.2.3" + is-stream "^2.0.0" + logform "^2.6.0" + one-time "^1.0.0" + readable-stream "^3.4.0" + safe-stable-stringify "^2.3.1" + stack-trace "0.0.x" + triple-beam "^1.3.0" + winston-transport "^4.7.0" + word-wrap@^1.2.3, word-wrap@~1.2.3: version "1.2.3" resolved "https://registry.yarnpkg.com/word-wrap/-/word-wrap-1.2.3.tgz#610636f6b1f703891bd34771ccb17fb93b47079c"