diff --git a/README.md b/README.md index 9d2141ab..53e3b035 100644 --- a/README.md +++ b/README.md @@ -14,17 +14,17 @@ This server provides a set of server-side services that are useful for the FHIR * [Package server](packages/readme.md) - **NPM-style FHIR package registry** with search, versioning, and downloads, consistent with the FHIR NPM Specification (as running at http://packages2.fhir.org/packages) * [XIG server](xig/readme.md) - **Comprehensive FHIR IG analytics** with resource breakdowns by version, authority, and realm (as running at http://packages2.fhir.org/packages) * [Publisher](publisher/readme.md) - FHIR publishing services (as running at [healthintersections.com.au](http://www.healthintersections.com.au/publisher)) -* [VCL](vcl/readme.md) - **Parse VCL expressions** into FHIR ValueSet resources (as running at http://fhir.org/vcl) +* [VCL](vcl/readme.md) - **Parse VCL expressions** into FHIR ValueSet resources (as running at http://fhir.org/vcl) * (Coming) Token services ## Summary Statement -* Maintainers: Grahame Grieve, Italo Macêdo, Josh Mandel, Jose Costa Teixeira +* Maintainers: Grahame Grieve, Italo Macêdo, Josh Mandel, Jose Costa Teixeira * Issues / Discussion: Use github issues * License: BSD-3 -* Contribution Policy: Make PRs. PRs have to pass all the tests +* Contribution Policy: Make PRs. PRs have to pass all the tests * Security Information: See [security.md](security.md) - + ## Build Status ![CI Build](https://github.com/HealthIntersections/fhirsmith/actions/workflows/ci.yml/badge.svg) [![Release](https://img.shields.io/github/v/release/HealthIntersections/fhirsmith?include_prereleases)](https://github.com/HealthIntersections/fhirsmith/releases) @@ -44,15 +44,15 @@ There are 4 executable programs: Unless you're developing, you only need the first two FHIRsmith is open source - see below, and you're welcome to use it for any kind of use. Note, -though, that if you support FHIRsmith commercially as part of a managed service or product, you +though, that if you support FHIRsmith commercially as part of a managed service or product, you are required to be a Commercial Partner of HL7 - see (link to be provided). ### Quick Start * Install FHIRSmith (using docker, or an NPM release, or just get the code by git) -* Figure out the data directory +* Figure out the data directory * Provide a configuration to tell the server what to run (see documentation below, or use a [prebuilt configuration]/configurations/readme.md) -* Run the server +* Run the server For further details of these steps, read on @@ -128,6 +128,36 @@ Create a `config.json` file in your data directory (use `config-template.json` a } ``` +### Logging Configuration + +Add a `logging` section to `config.json` to control log behaviour. All fields are optional and have sensible defaults: + +```json +{ + "logging": { + "level": "info", + "console": true, + "consoleErrors": false, + "maxFiles": 14, + "maxSize": "50m", + "flushInterval": 2000, + "flushSize": 200 + } +} +``` + +| Option | Default | Description | +|---|---|---| +| `level` | `"info"` | Minimum level to log: `error`, `warn`, `info`, `debug`, or `verbose` | +| `console` | `true` | Write log lines to stdout/stderr. Disable when running as a systemd service where console output goes to the journal and is redundant | +| `consoleErrors` | `false` | Whether `error` and `warn` levels appear on the console. When `false`, errors and warnings are written to the log file only | +| `maxFiles` | `14` | Number of daily log files to retain before old ones are deleted | +| `maxSize` | `0` (unlimited) | Maximum size per log file before rotation. Accepts human-readable strings: `"20m"`, `"1g"`, or a raw byte count | +| `flushInterval` | `2000` | Milliseconds between buffered writes to disk. Increase to reduce I/O under heavy load | +| `flushSize` | `200` | Number of buffered log lines that trigger an immediate flush regardless of the timer | + +Log files are written to the `logs/` subdirectory of the data directory as `server-YYYY-MM-DD.log`. A `server.log` symlink always points to the current day's file, so `tail -f data/logs/server.log` tracks the active log without needing to know the date. + ### Start the Server ```bash @@ -211,8 +241,8 @@ Available tags: ### Windows Installation -You can install as a windows service using [windows-install.js](utilities/windows-install.js). You might need to -hack that. +You can install as a windows service using [windows-install.js](utilities/windows-install.js). You might need to +hack that. ## Releases @@ -224,9 +254,9 @@ Each GitHub Release includes: - **Release notes** extracted from CHANGELOG.md - **Source code** archives (zip and tar.gz) - **Docker images** pushed to GitHub Container Registry: - - `ghcr.io/healthintersections/fhirsmith:latest` - - `ghcr.io/healthintersections/fhirsmith:vX.Y.Z` - - `ghcr.io/healthintersections/fhirsmith:X.Y.Z` + - `ghcr.io/healthintersections/fhirsmith:latest` + - `ghcr.io/healthintersections/fhirsmith:vX.Y.Z` + - `ghcr.io/healthintersections/fhirsmith:X.Y.Z` - **npm package** published to npmjs.org as `fhirsmith` *(if you add this)* ### Creating a Release @@ -244,13 +274,13 @@ GitHub Actions will automatically: 1. Update `CHANGELOG.md` with your changes under a new version section: ```markdown ## [vX.Y.Z] - YYYY-MM-DD - ### Added - - New feature description - ### Changed - - Change description - ### Fixed - - Bug fix description - ### Tx Conformance Statement +### Added +- New feature description +### Changed +- Change description +### Fixed +- Bug fix description +### Tx Conformance Statement {copy content from text-cases-summary.txt} ``` 2. Update `package.json` to have the same release version @@ -270,9 +300,9 @@ or do it via a PR ``` 5. Monitor the release: - - Check [GitHub Actions](https://github.com/HealthIntersections/fhirsmith/actions) for the Release workflow - - Verify the [GitHub Release](https://github.com/HealthIntersections/fhirsmith/releases) was created - - Confirm Docker images are available at [GHCR](https://github.com/HealthIntersections/fhirsmith/pkgs/container/fhirsmith) + - Check [GitHub Actions](https://github.com/HealthIntersections/fhirsmith/actions) for the Release workflow + - Verify the [GitHub Release](https://github.com/HealthIntersections/fhirsmith/releases) was created + - Confirm Docker images are available at [GHCR](https://github.com/HealthIntersections/fhirsmith/pkgs/container/fhirsmith) 6. Update `package.json` to have the next release version -SNAPSHOT diff --git a/extension-tracker/extension-tracker-template.html b/extension-tracker/extension-tracker-template.html index c25ff32b..4ddc5cb0 100644 --- a/extension-tracker/extension-tracker-template.html +++ b/extension-tracker/extension-tracker-template.html @@ -92,7 +92,9 @@

FHIR © HL7.org 2011+.  |  FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  |  - [%total-packages%] packages tracked  |  ([%ms%] ms) + [%total-packages%] packages tracked  | +   ([%ms%] ms) + [%sponsorMessage%]

diff --git a/library/html-server.js b/library/html-server.js index e3fa93cc..0864483d 100644 --- a/library/html-server.js +++ b/library/html-server.js @@ -9,6 +9,8 @@ const fs = require('fs'); const path = require('path'); const escape = require('escape-html'); +let sponsorMessage = ''; + class HtmlServer { log; @@ -20,6 +22,10 @@ class HtmlServer { this.log = logv; } + setSponsorMessage(msg) { + sponsorMessage = msg; + } + // Template Management loadTemplate(templateName, templatePath) { try { @@ -73,6 +79,7 @@ class HtmlServer { .replace(/\[%endpoint-path%\]/g, escape(renderOptions.endpointpath)) .replace(/\[%fhir-version%\]/g, escape(renderOptions.fhirversion)) .replace(/\[%ms%\]/g, escape(renderOptions.processingTime.toString())) + .replace(/\[%sponsorMessage%\]/g, sponsorMessage) .replace(/\[%about%\]/g, renderOptions.about || ''); // Handle any custom template variables diff --git a/library/logger-telnet.js b/library/logger-telnet.js deleted file mode 100644 index 2986441b..00000000 --- a/library/logger-telnet.js +++ /dev/null @@ -1,205 +0,0 @@ -/** - * Custom Winston transport for streaming logs over a TCP socket - * This allows viewing logs remotely via telnet or a custom client - */ -const winston = require('winston'); -const net = require('net'); -const Transport = winston.Transport; - -/** - * Winston transport that streams logs to connected TCP clients - * @extends {winston.Transport} - */ -class SocketTransport extends Transport { - /** - * Create a new socket transport - * @param {Object} options - Transport options - * @param {string} [options.host='127.0.0.1'] - Host to bind to - * @param {number} [options.port=9300] - Port to listen on - * @param {string} [options.level='info'] - Minimum log level to stream - * @param {Function} [options.format] - Custom formatter function (msg, level, meta) => string - */ - constructor(options = {}) { - super(options); - this.name = 'socket'; - this.level = options.level || 'info'; - this.clients = new Set(); - this.format = options.format || this._defaultFormat; - - // Create TCP server - this.server = net.createServer((socket) => { - console.log(`Client connected to log stream from ${socket.remoteAddress}`); - - // Send welcome message - socket.write(`=== Connected to log stream (${new Date().toISOString()}) ===\n`); - socket.write(`=== Log level: ${this.level} ===\n\n`); - - // Add to clients set - this.clients.add(socket); - - socket.on('close', () => { - console.log(`Client disconnected from log stream: ${socket.remoteAddress}`); - this.clients.delete(socket); - }); - - socket.on('error', (err) => { - console.error(`Socket error: ${err.message}`); - this.clients.delete(socket); - }); - - // Support for simple commands - socket.on('data', (data) => { - const command = data.toString().trim().toLowerCase(); - - if (command === 'help') { - socket.write(this._getHelpText()); - } else if (command === 'stats') { - socket.write(this._getStatsText()); - } else if (command === 'quit' || command === 'exit') { - socket.end('=== Disconnected ===\n'); - } else if (command.startsWith('level ')) { - const newLevel = command.split(' ')[1]; - if (['error', 'warn', 'info', 'debug', 'verbose', 'silly'].includes(newLevel)) { - this.level = newLevel; - socket.write(`=== Log level changed to ${newLevel} ===\n`); - } else { - socket.write(`=== Invalid log level: ${newLevel} ===\n`); - } - } - }); - }); - - // Start listening - const port = options.port || 9300; - const host = options.host || '127.0.0.1'; - - this.server.listen(port, host, () => { - console.log(`Log socket server running on ${host}:${port}`); - }); - - this.server.on('error', (err) => { - console.error(`Socket transport error: ${err.message}`); - }); - } - - /** - * Winston transport method called for each log - * @param {Object} info - Log information - * @param {Function} callback - Callback function - */ - log(info, callback) { - setImmediate(() => { - this.emit('logged', info); - }); - - // Skip if no clients connected - if (this.clients.size === 0) { - callback(); - return; - } - - // Format the log entry - const logEntry = this.format(info); - - // Send to all connected clients - const deadClients = []; - for (const client of this.clients) { - try { - client.write(logEntry); - } catch (err) { - deadClients.push(client); - } - } - - // Remove dead connections - deadClients.forEach(client => this.clients.delete(client)); - - callback(); - } - - /** - * Default log formatter - * @param {Object} info - Log information - * @returns {string} Formatted log entry - * @private - */ - _defaultFormat(info) { - const timestamp = info.timestamp || new Date().toISOString(); - const level = info.level.toUpperCase().padEnd(7); - const message = info.message || ''; - - // Extract metadata excluding standard fields - const meta = { ...info }; - delete meta.timestamp; - delete meta.level; - delete meta.message; - - const metaStr = Object.keys(meta).length > 0 - ? ` ${JSON.stringify(meta)}` - : ''; - - return `${timestamp} [${level}] ${message}${metaStr}\n`; - } - - /** - * Get help text for connected clients - * @returns {string} Help text - * @private - */ - _getHelpText() { - return ` -=== Log Viewer Commands === -help - Show this help -stats - Show connection statistics -level - Change log level (error, warn, info, debug, verbose, silly) -quit/exit - Disconnect - -=== Log Format === -TIMESTAMP [LEVEL] MESSAGE {metadata} - -=== Examples === -level debug - Show debug and higher priority logs -level error - Show only error logs - -`.trim() + '\n\n'; - } - - /** - * Get statistics text for connected clients - * @returns {string} Statistics text - * @private - */ - _getStatsText() { - return ` -=== Log Statistics === -Connected clients: ${this.clients.size} -Current log level: ${this.level} -Server started: ${this.server.startTime || 'unknown'} -Current time: ${new Date().toISOString()} - -`.trim() + '\n\n'; - } - - /** - * Close the socket server - * @param {Function} [callback] - Callback function - */ - close(callback) { - // Notify all clients - for (const client of this.clients) { - try { - client.end('=== Log server shutting down ===\n'); - } catch (err) { - // Ignore errors during shutdown - } - } - - // Close server - this.server.close(callback); - } -} - -// Register the transport with Winston -winston.transports.Socket = SocketTransport; - -module.exports = SocketTransport; diff --git a/library/logger.js b/library/logger.js index 1154b4ac..3ddf04b5 100644 --- a/library/logger.js +++ b/library/logger.js @@ -1,8 +1,25 @@ -const winston = require('winston'); -require('winston-daily-rotate-file'); const fs = require('fs'); +const path = require('path'); const folders = require('./folder-setup'); +// --------------------------------------------------------------------------- +// Buffered, daily-rotating logger +// - Writes are batched and flushed every FLUSH_INTERVAL ms or FLUSH_SIZE lines +// - this is intended to be highly efficient +// --------------------------------------------------------------------------- + +const DEFAULTS = { + level: 'info', // error, warn, info, debug, verbose + console: true, // write to stdout/stderr + consoleErrors: false, // include error/warn on console (when running as service, these go to journal) + maxFiles: 14, // number of daily log files to keep + maxSize: 0, // max bytes per file (0 = unlimited) + flushInterval: 2000, // ms between flushes + flushSize: 200, // flush when buffer reaches this many lines +}; + +const LEVELS = { error: 0, warn: 1, info: 2, debug: 3, verbose: 4 }; + class Logger { static _instance = null; @@ -13,260 +30,278 @@ class Logger { return Logger._instance; } + // options = config.logging section from config.json (all fields optional) + // + // Example config.json: + // { + // "logging": { + // "level": "info", + // "console": false, + // "consoleErrors": false, + // "maxFiles": 14, + // "maxSize": "50m", + // "flushInterval": 2000, + // "flushSize": 200 + // } + // } + // constructor(options = {}) { - this.options = { - level: options.level || 'info', - logDir: options.logDir || folders.logsDir(), - console: options.console !== undefined ? options.console : true, - consoleErrors: options.consoleErrors !== undefined ? options.consoleErrors : false, - telnetErrors: options.telnetErrors !== undefined ? options.telnetErrors : false, - file: { - filename: options.file?.filename || 'server-%DATE%.log', - datePattern: options.file?.datePattern || 'YYYY-MM-DD', - maxSize: options.file?.maxSize || '20m', - maxFiles: options.file?.maxFiles || 14 - } - }; + this.level = options.level || DEFAULTS.level; + this.logDir = options.logDir || folders.logsDir(); + this.maxFiles = options.maxFiles ?? DEFAULTS.maxFiles; + this.maxSize = Logger._parseSize(options.maxSize) || DEFAULTS.maxSize; + this.showConsole = options.console ?? DEFAULTS.console; + this.consoleErrors = options.consoleErrors ?? DEFAULTS.consoleErrors; + + const flushInterval = options.flushInterval ?? DEFAULTS.flushInterval; + this._flushSize = options.flushSize ?? DEFAULTS.flushSize; // Ensure log directory exists - if (!fs.existsSync(this.options.logDir)) { - fs.mkdirSync(this.options.logDir, { recursive: true }); + if (!fs.existsSync(this.logDir)) { + fs.mkdirSync(this.logDir, { recursive: true }); } - // Telnet clients storage - this.telnetClients = new Set(); + // Buffer + this._buffer = []; + this._currentDate = null; + this._fd = null; + this._currentFileSize = 0; - this._createLogger(); + // Periodic flush + this._flushTimer = setInterval(() => this._flush(), flushInterval); + if (this._flushTimer.unref) this._flushTimer.unref(); // don't keep process alive - // Log logger initialization - this.info('Logger initialized @ ' + this.options.logDir, { - level: this.options.level, - logDir: this.options.logDir - }); + // Flush on exit + process.on('exit', () => this._flushSync()); + + this.info('Logger initialized @ ' + this.logDir, {}); } - _createLogger() { - // Define formats for file output (with full metadata including stack traces) - const fileFormats = [ - winston.format.timestamp({ format: 'YYYY-MM-DD HH:mm:ss.SSS' }), - winston.format.errors({ stack: true }), - winston.format.splat(), - winston.format.json() - ]; - - // Create transports - const transports = []; - - // Add file transport with rotation (includes ALL levels with full metadata) - const fileTransport = new winston.transports.DailyRotateFile({ - dirname: this.options.logDir, - filename: this.options.file.filename, - datePattern: this.options.file.datePattern, - maxSize: this.options.file.maxSize, - maxFiles: this.options.file.maxFiles, - level: this.options.level, - format: winston.format.combine(...fileFormats) - }); - transports.push(fileTransport); - - // Add console transport if enabled - if (this.options.console) { - const consoleFormat = winston.format.combine( - winston.format.timestamp({ format: 'YYYY-MM-DD HH:mm:ss.SSS' }), - winston.format.errors({ stack: true }), - winston.format.colorize({ all: true }), - winston.format.printf(info => { - const stack = info.stack ? `\n${info.stack}` : ''; - return `${info.timestamp} ${info.level.padEnd(7)} ${info.message}${stack}`; - }) - ); - - const consoleTransport = new winston.transports.Console({ - level: this.options.level, - format: consoleFormat - }); - - transports.push(consoleTransport); + // Parse human-readable size strings: "20m" -> bytes, "1g" -> bytes + static _parseSize(value) { + if (!value) return 0; + if (typeof value === 'number') return value; + const m = String(value).match(/^(\d+(?:\.\d+)?)\s*([kmg])?b?$/i); + if (!m) return 0; + const num = parseFloat(m[1]); + switch ((m[2] || '').toLowerCase()) { + case 'k': return num * 1024; + case 'm': return num * 1024 * 1024; + case 'g': return num * 1024 * 1024 * 1024; + default: return num; } + } - // Create the winston logger - this.logger = winston.createLogger({ - level: this.options.level, - transports, - exitOnError: false - }); + // Compatibility: server.js home page reads Logger.getInstance().options.file.maxFiles etc. + get options() { + return { + level: this.level, + file: { + maxFiles: this.maxFiles, + maxSize: this.maxSize > 0 ? `${Math.round(this.maxSize / 1024 / 1024)}m` : 'unlimited', + } + }; } - // Telnet client management - addTelnetClient(socket) { - this.telnetClients.add(socket); + // --- formatting (inline, no libraries) --- + + _timestamp() { + const d = new Date(); + const Y = d.getFullYear(); + const M = String(d.getMonth() + 1).padStart(2, '0'); + const D = String(d.getDate()).padStart(2, '0'); + const h = String(d.getHours()).padStart(2, '0'); + const m = String(d.getMinutes()).padStart(2, '0'); + const s = String(d.getSeconds()).padStart(2, '0'); + const ms = String(d.getMilliseconds()).padStart(3, '0'); + return `${Y}-${M}-${D} ${h}:${m}:${s}.${ms}`; } - removeTelnetClient(socket) { - this.telnetClients.delete(socket); + _dateTag() { + const d = new Date(); + const Y = d.getFullYear(); + const M = String(d.getMonth() + 1).padStart(2, '0'); + const D = String(d.getDate()).padStart(2, '0'); + return `${Y}-${M}-${D}`; } - _sendToTelnet(level, message, stack, options) { - // Check if we should send errors/warnings to telnet - if (!options.telnetErrors && (level === 'error' || level === 'warn')) { - return; - } + _formatLine(level, message, stack) { + const ts = this._timestamp(); + const lv = level.padEnd(7); + let line = `${ts} ${lv} ${message}\n`; + if (stack) line += stack + '\n'; + return line; + } + + // --- file management --- - const timestamp = new Date().toISOString().replace('T', ' ').substring(0, 23); - let line = `${timestamp} ${level.padEnd(7)} ${message}\n`; - if (stack) { - line += stack + '\n'; + _openFile(dateTag) { + // Check if we need to rotate due to size + if (this._fd !== null && this._currentDate === dateTag) { + if (this.maxSize <= 0 || this._currentFileSize < this.maxSize) return; + // Size limit exceeded — close and fall through to open a new file + try { fs.closeSync(this._fd); } catch (_) { /* intentional */ } + this._fd = null; } + // Close previous (date changed) + if (this._fd !== null) { + try { fs.closeSync(this._fd); } catch (_) { /* intentional */ } + } + const filename = `server-${dateTag}.log`; + const filePath = path.join(this.logDir, filename); + this._fd = fs.openSync(filePath, 'a'); + try { this._currentFileSize = fs.fstatSync(this._fd).size; } catch (_) { this._currentFileSize = 0; } + this._currentDate = dateTag; + + // Maintain a stable symlink so `tail -f server.log` always tracks the current file + const linkPath = path.join(this.logDir, 'server.log'); + try { fs.unlinkSync(linkPath); } catch (_) { /* intentional */ } + try { fs.symlinkSync(filename, linkPath); } catch (_) { /* intentional */ } + + this._purgeOldFiles(); + } - for (const client of this.telnetClients) { - try { - client.write(line); - } catch (e) { - // Client disconnected, remove it - this.telnetClients.delete(client); + _purgeOldFiles() { + try { + const files = fs.readdirSync(this.logDir) + .filter(f => f.startsWith('server-') && f.endsWith('.log')) + .sort(); + while (files.length > this.maxFiles) { + const old = files.shift(); + fs.unlinkSync(path.join(this.logDir, old)); } - } + } catch (_) { /* intentional */ } } - _shouldLogToConsole(level, options) { - if (level === 'error' || level === 'warn') { - return options.consoleErrors; + // --- buffer + flush --- + + _enqueue(line) { + this._buffer.push(line); + if (this._buffer.length >= this._flushSize) { + this._flush(); } - return true; + } + + _flush() { + if (this._buffer.length === 0) return; + const dateTag = this._dateTag(); + this._openFile(dateTag); + const chunk = this._buffer.join(''); + this._buffer.length = 0; + // Async write — fire and forget; OS will buffer anyway + const buf = Buffer.from(chunk); + this._currentFileSize += buf.length; + fs.write(this._fd, buf, 0, buf.length, null, (err) => { + if (err) { + // If the fd went bad (e.g. date rolled), reopen and retry once + try { + this._currentDate = null; + this._openFile(this._dateTag()); + fs.writeSync(this._fd, buf, 0, buf.length); + } catch (_) { /* intentional */ } + } + }); + } + + _flushSync() { + if (this._buffer.length === 0) return; + const dateTag = this._dateTag(); + this._openFile(dateTag); + const chunk = this._buffer.join(''); + this._buffer.length = 0; + try { fs.writeSync(this._fd, chunk); } catch (_) { /* intentional */ } + } + + // --- core log --- + + _shouldLog(level) { + return (LEVELS[level] ?? 99) <= (LEVELS[this.level] ?? 2); } _log(level, messageOrError, meta, options) { + if (!this._shouldLog(level)) return; + let message; let stack; - // Check if we should skip console for errors/warnings - const skipConsole = !this._shouldLogToConsole(level, options); - - // Handle Error objects if (messageOrError instanceof Error) { message = messageOrError.message; stack = messageOrError.stack; - if (skipConsole) { - // Log only to file transport - this.logger.transports - .filter(t => !(t instanceof winston.transports.Console)) - .forEach(t => t.log({ level, message, stack, ...meta })); - } else { - this.logger[level](message, {stack, ...meta}); - } } else { message = String(messageOrError); stack = meta?.stack; - if (skipConsole) { - this.logger.transports - .filter(t => !(t instanceof winston.transports.Console)) - .forEach(t => t.log({ level, message, ...meta })); - } else { - this.logger[level](message, meta); - } } - this._sendToTelnet(level, message, stack, options); - } - - error(message, meta = {}) { - this._log('error', message, meta, this.options); - } - - warn(message, meta = {}) { - this._log('warn', message, meta, this.options); - } - - info(message, meta = {}) { - this._log('info', message, meta, this.options); + const line = this._formatLine(level, message, stack); + + // Buffer for file + this._enqueue(line); + + // Console + if (this.showConsole) { + const isErrWarn = level === 'error' || level === 'warn'; + const consoleErrors = options?.consoleErrors ?? this.consoleErrors; + if (!isErrWarn || consoleErrors) { + if (isErrWarn) { + process.stderr.write(line); + } else { + process.stdout.write(line); + } + } + } } - debug(message, meta = {}) { - this._log('debug', message, meta, this.options); - } + // --- public API (same as before) --- - verbose(message, meta = {}) { - this._log('verbose', message, meta, this.options); - } + error(message, meta = {}) { this._log('error', message, meta, this); } + warn(message, meta = {}) { this._log('warn', message, meta, this); } + info(message, meta = {}) { this._log('info', message, meta, this); } + debug(message, meta = {}) { this._log('debug', message, meta, this); } + verbose(message, meta = {}) { this._log('verbose', message, meta, this); } - log(level, message, meta = {}) { - this._log(level, message, meta, this.options); - } + log(level, message, meta = {}) { this._log(level, message, meta, this); } child(defaultMeta = {}) { const self = this; - // Build module-specific options const childOptions = { - consoleErrors: defaultMeta.consoleErrors ?? self.options.consoleErrors, - telnetErrors: defaultMeta.telnetErrors ?? self.options.telnetErrors + consoleErrors: defaultMeta.consoleErrors ?? self.consoleErrors, }; - // Remove our custom options from defaultMeta so they don't pollute log output - const cleanMeta = { ...defaultMeta }; - delete cleanMeta.consoleErrors; - delete cleanMeta.telnetErrors; - - if (cleanMeta.module) { - const modulePrefix = `{${cleanMeta.module}}`; - - return { - error: (messageOrError, meta = {}) => { - if (messageOrError instanceof Error) { - const prefixedError = new Error(`${modulePrefix}: ${messageOrError.message}`); - prefixedError.stack = messageOrError.stack; - self._log('error', prefixedError, meta, childOptions); - } else { - self._log('error', `${modulePrefix}: ${messageOrError}`, meta, childOptions); - } - }, - warn: (messageOrError, meta = {}) => { - if (messageOrError instanceof Error) { - const prefixedError = new Error(`${modulePrefix}: ${messageOrError.message}`); - prefixedError.stack = messageOrError.stack; - self._log('warn', prefixedError, meta, childOptions); - } else { - self._log('warn', `${modulePrefix}: ${messageOrError}`, meta, childOptions); - } - }, - info: (message, meta = {}) => self._log('info', `${modulePrefix}: ${message}`, meta, childOptions), - debug: (message, meta = {}) => self._log('debug', `${modulePrefix}: ${message}`, meta, childOptions), - verbose: (message, meta = {}) => self._log('verbose', `${modulePrefix}: ${message}`, meta, childOptions), - log: (level, message, meta = {}) => self._log(level, `${modulePrefix}: ${message}`, meta, childOptions) - }; - } + const modulePrefix = defaultMeta.module ? `{${defaultMeta.module}}` : null; - // For other metadata without module prefix - const childLogger = { - error: (messageOrError, meta = {}) => self._log('error', messageOrError, { ...cleanMeta, ...meta }, childOptions), - warn: (messageOrError, meta = {}) => self._log('warn', messageOrError, { ...cleanMeta, ...meta }, childOptions), - info: (message, meta = {}) => self._log('info', message, { ...cleanMeta, ...meta }, childOptions), - debug: (message, meta = {}) => self._log('debug', message, { ...cleanMeta, ...meta }, childOptions), - verbose: (message, meta = {}) => self._log('verbose', message, { ...cleanMeta, ...meta }, childOptions), - log: (level, message, meta = {}) => self._log(level, message, { ...cleanMeta, ...meta }, childOptions) + const wrap = (level) => (messageOrError, meta = {}) => { + if (messageOrError instanceof Error) { + const prefixed = modulePrefix + ? Object.assign(new Error(`${modulePrefix}: ${messageOrError.message}`), { stack: messageOrError.stack }) + : messageOrError; + self._log(level, prefixed, meta, childOptions); + } else { + const msg = modulePrefix ? `${modulePrefix}: ${messageOrError}` : String(messageOrError); + self._log(level, msg, meta, childOptions); + } }; - return childLogger; + return { + error: wrap('error'), + warn: wrap('warn'), + info: wrap('info'), + debug: wrap('debug'), + verbose: wrap('verbose'), + log: (level, message, meta = {}) => wrap(level)(message, meta) + }; } setLevel(level) { - this.options.level = level; - this.logger.transports.forEach(transport => { - transport.level = level; - }); + this.level = level; this.info(`Log level changed to ${level}`); } setConsoleErrors(enabled) { - this.options.consoleErrors = enabled; + this.consoleErrors = enabled; this.info(`Console errors ${enabled ? 'enabled' : 'disabled'}`); } - setTelnetErrors(enabled) { - this.options.telnetErrors = enabled; - this.info(`Telnet errors ${enabled ? 'enabled' : 'disabled'}`); - } - stream() { return { write: (message) => { @@ -274,6 +309,11 @@ class Logger { } }; } + + // Force an immediate flush (e.g. before graceful shutdown) + flush() { + this._flushSync(); + } } module.exports = Logger; \ No newline at end of file diff --git a/package.json b/package.json index 9ec0def7..553abc50 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,7 @@ { "name": "fhirsmith", - "version": "0.8.5", + "version": "0.8.6-SNAPSHOT", + "txVersion" : "1.9.1", "description": "A Node.js server that provides a collection of tools to serve the FHIR ecosystem", "main": "server.js", "engines": { diff --git a/packages/packages-template.html b/packages/packages-template.html index 7967016d..f6e9fb2a 100644 --- a/packages/packages-template.html +++ b/packages/packages-template.html @@ -92,7 +92,9 @@

FHIR © HL7.org 2011+.  |  FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  |  - Package Registry last updated as of [%crawler-date%]  |  [%total-packages%] packages  |  ([%ms%] ms) + Package Registry last updated as of [%crawler-date%]  |  [%total-packages%] packages  | +   ([%ms%] ms) + [%sponsorMessage%]

diff --git a/publisher/publisher-template.html b/publisher/publisher-template.html index e45a77f5..baa75960 100644 --- a/publisher/publisher-template.html +++ b/publisher/publisher-template.html @@ -123,6 +123,7 @@

FHIR © HL7.org 2011+.  |  FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  |  ([%ms%] ms) + [%sponsorMessage%]

diff --git a/registry/registry-template.html b/registry/registry-template.html index f640b1ea..18229544 100644 --- a/registry/registry-template.html +++ b/registry/registry-template.html @@ -92,7 +92,9 @@

FHIR © HL7.org 2011+.  |  FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  |  - Terminology Registry last updated as of [%crawler-date%]  |  [%total-packages%] packages  |  ([%ms%] ms) + Terminology Registry last updated as of [%crawler-date%]  |  [%total-packages%] packages  | +   ([%ms%] ms) + [%sponsorMessage%]

diff --git a/root-template.html b/root-template.html index 284c91f4..842eca69 100644 --- a/root-template.html +++ b/root-template.html @@ -90,8 +90,9 @@

FHIR © HL7.org 2011+.  |  - FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  |  ([%ms%] ms) -   + FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  | +   ([%ms%] ms) + [%sponsorMessage%]

diff --git a/server.js b/server.js index 7a26a80e..c554e4c1 100644 --- a/server.js +++ b/server.js @@ -28,7 +28,7 @@ try { } const Logger = require('./library/logger'); -const serverLog = Logger.getInstance().child({ module: 'server' }); +const serverLog = Logger.getInstance(config.logging || {}).child({ module: 'server' }); const packageJson = require('./package.json'); // Startup banner @@ -64,6 +64,7 @@ const FolderModule = require("./folder/folder"); const ExtensionTrackerModule = require("./extension-tracker/extension-tracker"); htmlServer.useLog(serverLog); +htmlServer.setSponsorMessage(config.sponsorMessage ? config.sponsorMessage : ''); const app = express(); @@ -386,22 +387,44 @@ async function buildRootPageContent() { // Memory usage const memUsage = process.memoryUsage(); - const heapUsedMB = (memUsage.heapUsed / 1024 / 1024).toFixed(2); - const heapAvailableMB = ((memUsage.heapTotal - memUsage.heapUsed) / 1024 / 1024).toFixed(2); - const rssMB = (memUsage.rss / 1024 / 1024).toFixed(2); - const freeMemMB = (os.freemem() / 1024 / 1024).toFixed(0); + const heapStats = v8.getHeapStatistics(); + + // V8 heap: used vs limit + const v8UsedMB = (memUsage.heapUsed / 1024 / 1024).toFixed(0); + const v8LimitMB = (heapStats.heap_size_limit / 1024 / 1024).toFixed(0); + const v8PCT = (memUsage.heapUsed * 100) / heapStats.heap_size_limit; + + // Process RSS vs cgroup limit (or system total) + const rssMB = (memUsage.rss / 1024 / 1024).toFixed(0); + let memLimit; + try { + const raw = fs.readFileSync('/sys/fs/cgroup/memory.max', 'utf8').trim(); + memLimit = raw === 'max' ? os.totalmem() : parseInt(raw); + } catch { + memLimit = os.totalmem(); + } + const memLimitMB = (memLimit / 1024 / 1024).toFixed(0); + const processPCT = (memUsage.rss * 100) / memLimit; + + // System memory const totalMemMB = (os.totalmem() / 1024 / 1024).toFixed(0); + const usedMemMB = ((os.totalmem() - os.freemem()) / 1024 / 1024).toFixed(0); + const sysMemPCT = ((os.totalmem() - os.freemem()) * 100) / os.totalmem(); + + // Average requests per minute + const uptimeMinutesTotal = uptimeMs / 60000; + const avgReqPerMin = uptimeMinutesTotal > 0 ? (stats.requestCount / uptimeMinutesTotal).toFixed(1) : '0.0'; content += ''; content += ''; content += ``; content += ``; - content += ``; + content += ``; content += ''; content += ''; - content += ``; - content += ``; - content += ``; + content += ``; + content += ``; + content += ``; content += ''; content += getLogStats(); content += '
Uptime: ${escape(uptimeStr)}Request Count: ${stats.requestCount} (static: ${stats.staticRequestCount})Free Memory: ${freeMemMB} MB of ${totalMemMB} MBAvg Requests/min: ${avgReqPerMin}
Heap Used: ${heapUsedMB} MBHeap Available: ${heapAvailableMB} MBProcess Memory: ${rssMB} MBV8 Memory: ${v8UsedMB} MB of ${v8LimitMB} MB (${v8PCT.toFixed(0)}%)Process Memory: ${rssMB} MB of ${memLimitMB} MB (${processPCT.toFixed(0)}%)System Memory: ${usedMemMB} MB of ${totalMemMB} MB (${sysMemPCT.toFixed(0)}%)
'; @@ -543,10 +566,24 @@ app.get('/dashboard', async (req, res) => { // Memory usage const memUsage = process.memoryUsage(); const heapStats = v8.getHeapStatistics(); - const nodeMemPCT = (memUsage.heapUsed * 100) / heapStats.heap_size_limit; // % of Node.js memory limit used + const v8PCT = (memUsage.heapUsed * 100) / heapStats.heap_size_limit; // % of V8 heap limit used + + // Process RSS as % of cgroup memory limit (or system total as fallback) + let memLimit; + try { + const raw = fs.readFileSync('/sys/fs/cgroup/memory.max', 'utf8').trim(); + memLimit = raw === 'max' ? os.totalmem() : parseInt(raw); + } catch { + memLimit = os.totalmem(); + } + const processPCT = (memUsage.rss * 100) / memLimit; + const totalMemBytes = os.totalmem(); const freeMemBytes = os.freemem(); const sysMemPCT = ((totalMemBytes - freeMemBytes) * 100) / totalMemBytes; // % of system memory used + + const memMaxPCT = Math.max(v8PCT, processPCT, sysMemPCT); + const fstats = fs.statfsSync(folders.logsDir()); const diskPCT = 100 - ((fstats.bavail * 100) / fstats.blocks); // % of disk used @@ -555,8 +592,7 @@ app.get('/dashboard', async (req, res) => { content += ''; content += `Uptime: ${escape(uptimeStr)}`; content += `Request Count: ${stats.requestCount} (static: ${stats.staticRequestCount})`; - content += `Node Memory: ${nodeMemPCT.toFixed(0)}%`; - content += `System Memory: ${sysMemPCT.toFixed(0)}%`; + content += `Memory: ${v8PCT.toFixed(0)}% V8, ${processPCT.toFixed(0)}% Process, ${sysMemPCT.toFixed(0)}% System`; content += `Disk: ${diskPCT.toFixed(0)}%`; content += ''; content += ''; diff --git a/translations/Messages.properties b/translations/Messages.properties index 283fdc36..38d2391f 100644 --- a/translations/Messages.properties +++ b/translations/Messages.properties @@ -1274,7 +1274,7 @@ VALUESET_SHAREABLE_MISSING = Published value sets SHOULD conform to the Shareabl VALUESET_SHAREABLE_MISSING_HL7 = Value sets published by HL7 SHALL conform to the ShareableValueSet profile, which says that the element ValueSet.{0} is mandatory, but it is not present VALUESET_SUPPLEMENT_MISSING_one = Required supplement not found: {1} VALUESET_SUPPLEMENT_MISSING_other = Required supplements not found: {1} -VALUESET_TOO_COSTLY = The value set ''{0}'' expansion has too many codes to display ({1}) +VALUESET_TOO_COSTLY = The value set ''{0}'' expansion has too many codes to produce ({1}) VALUESET_TOO_COSTLY_COUNT = The value set ''{0}'' expansion has {2} codes, which is too many to display ({1}) VALUESET_TOO_COSTLY_TIME = The value set ''{0}'' {2} took too long to process (>{1}sec) VALUESET_UNC_SYSTEM_WARNING = Unknown System ''{0}'' specified, so Concepts and Filters can''t be checked (Details: {1}) diff --git a/translations/rendering-phrases.properties b/translations/rendering-phrases.properties index c193de9d..0afbcf77 100644 --- a/translations/rendering-phrases.properties +++ b/translations/rendering-phrases.properties @@ -1214,4 +1214,6 @@ CONSENT_HT_RESOURCE_TYPE = Resource Type CONSENT_HT_DOC_TYPE = Document Type CONSENT_HT_CODE = Code WEB_SOURCE = Web Source -GENERAL_COPY = Click to Copy \ No newline at end of file +GENERAL_COPY = Click to Copy +CODESYSTEM_LVL = Level +FEATURE = Feature \ No newline at end of file diff --git a/tx/cs/cs-api.js b/tx/cs/cs-api.js index fa5a138c..5bfa9e20 100644 --- a/tx/cs/cs-api.js +++ b/tx/cs/cs-api.js @@ -719,7 +719,11 @@ class CodeSystemProvider { return false; } + hasMultiHierarchy() { + return false; + } /** + * * @returns {string} valueset for the code system */ valueSet() { diff --git a/tx/cs/cs-cs.js b/tx/cs/cs-cs.js index f5177ff4..8ac2c669 100644 --- a/tx/cs/cs-cs.js +++ b/tx/cs/cs-cs.js @@ -1521,7 +1521,11 @@ class FhirCodeSystemProvider extends BaseCSServices { } versionNeeded() { - return this.codeSystem.jsonObj.versionNeeded; + return this.codeSystem.jsonObj.versionNeeded ? true : false; + } + + hasMultiHierarchy() { + return this.codeSystem.hasMultiHierarchy; } } diff --git a/tx/cs/cs-snomed.js b/tx/cs/cs-snomed.js index 52f6ccc5..92472224 100644 --- a/tx/cs/cs-snomed.js +++ b/tx/cs/cs-snomed.js @@ -1348,6 +1348,10 @@ class SnomedProvider extends BaseCSServices { return result; } + hasMultiHierarchy() { + return true; + } + } /** diff --git a/tx/html/tx-template.html b/tx/html/tx-template.html index 59024141..949404ec 100644 --- a/tx/html/tx-template.html +++ b/tx/html/tx-template.html @@ -117,9 +117,10 @@

- FHIR © HL7.org 2011+.  |  + FHIR v[%fhir-version%] © HL7.org 2011+.  |  FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  |  - FHIR Version [%fhir-version%]  |  ([%ms%] ms) +   ([%ms%] ms) + [%sponsorMessage%]

diff --git a/tx/library/codesystem.js b/tx/library/codesystem.js index f6ec2a03..29354e85 100644 --- a/tx/library/codesystem.js +++ b/tx/library/codesystem.js @@ -74,6 +74,8 @@ class CodeSystem extends CanonicalResource { */ childToParentsMap = new Map(); + hasMultiHierarchy = false; + /** * Static factory method for convenience * @param {string} jsonString - JSON string representation of CodeSystem @@ -326,6 +328,8 @@ class CodeSystem extends CanonicalResource { // Third pass: handle nested concept structures this._buildNestedHierarchy(this.jsonObj.concept); + + this.hasMultiHierarchy = Array.from(this.childToParentsMap.values()).some(parents => parents.length > 1); } /** diff --git a/tx/library/renderer.js b/tx/library/renderer.js index 4eaadad1..9b0c69d8 100644 --- a/tx/library/renderer.js +++ b/tx/library/renderer.js @@ -133,7 +133,7 @@ class Renderer { return result; } - async renderMetadataTable(res, tbl) { + async renderMetadataTable(res, tbl, sourcePackage) { this.renderMetadataVersion(res, tbl); await this.renderMetadataProfiles(res, tbl); this.renderMetadataTags(res, tbl); @@ -154,6 +154,11 @@ class Renderer { this.renderProperty(tbl, 'EXT_FMM_LEVEL', Extensions.readString(res, 'http://hl7.org/fhir/StructureDefinition/structuredefinition-fmm')); this.renderProperty(tbl, 'PAT_PERIOD', res.effectivePeriod); this.renderPropertyLink(tbl, 'WEB_SOURCE', Extensions.readString(res, 'http://hl7.org/fhir/StructureDefinition/web-source')); + this.renderProperty(tbl, 'GENERAL_SOURCE', sourcePackage); + for (let ext of Extensions.list(res, 'http://hl7.org/fhir/uv/application-feature/StructureDefinition/feature')) { + this.renderProperty(tbl, 'FEATURE', this.featureSummary(ext)); + } + // capability statement things this.renderProperty(tbl, 'Kind', res.kind); @@ -165,7 +170,6 @@ class Renderer { } } this.renderProperty(tbl, 'GENERAL_URL', res.implementation?.url); - this.renderProperty(tbl, 'Kind', res.kind); this.renderProperty(tbl, 'EX_SCEN_FVER', res.fhirVersion); if (res.content === 'supplement' && res.supplements) { @@ -364,7 +368,7 @@ class Renderer { return div_.toString(); } - async renderCodeSystem(cs) { + async renderCodeSystem(cs, sourcePackage) { if (cs.json) { cs = cs.json; } @@ -373,7 +377,7 @@ class Renderer { // Metadata table div_.h3().tx("Properties"); - await this.renderMetadataTable(cs, div_.table("grid")); + await this.renderMetadataTable(cs, div_.table("grid"), sourcePackage); // Code system properties const hasProps = this.generateProperties(div_, cs); @@ -2227,6 +2231,18 @@ class Renderer { } } + featureSummary(ext) { + let defn = Extensions.readString(ext, 'definition'); + let value = Extensions.readString(ext, 'value'); + switch (defn) { + case 'http://hl7.org/fhir/uv/tx-tests/FeatureDefinition/test-version': + return 'Tx Test version = ' + value; + case 'http://hl7.org/fhir/uv/tx-ecosystem/FeatureDefinition/CodeSystemAsParameter': + return 'CodeSystems as parameters = ' + value; + default: + return defn+' = ' + value; + } + } } module.exports = { Renderer }; diff --git a/tx/operation-context.js b/tx/operation-context.js index f281778d..5ce14df0 100644 --- a/tx/operation-context.js +++ b/tx/operation-context.js @@ -15,7 +15,7 @@ function isDebugging() { } // Also check for debug flags in case inspector not yet attached return process.execArgv.some(arg => - arg.includes('--inspect') || arg.includes('--debug') + arg.includes('--inspect') || arg.includes('--debug') ); } @@ -233,16 +233,16 @@ class ExpansionCache { // Resources are now CodeSystem/ValueSet wrappers, not raw JSON if (additionalResources && additionalResources.length > 0) { const resourceHashes = additionalResources - .map(r => { - // Get the JSON object from wrapper or use directly - const json = r.jsonObj || r; - // Create a content hash for this resource - return crypto.createHash('sha256') - .update(JSON.stringify(json)) - .digest('hex') - .substring(0, 16); // Use first 16 chars for brevity - }) - .sort(); + .map(r => { + // Get the JSON object from wrapper or use directly + const json = r.jsonObj || r; + // Create a content hash for this resource + return crypto.createHash('sha256') + .update(JSON.stringify(json)) + .digest('hex') + .substring(0, 16); // Use first 16 chars for brevity + }) + .sort(); keyParts.push(`additional:${resourceHashes.join(',')}`); } @@ -314,7 +314,7 @@ class ExpansionCache { // Get entries sorted by lastUsed (oldest first) const entries = Array.from(this.cache.entries()) - .sort((a, b) => a[1].lastUsed - b[1].lastUsed); + .sort((a, b) => a[1].lastUsed - b[1].lastUsed); const toEvict = Math.min(count, entries.length); for (let i = 0; i < toEvict; i++) { @@ -413,7 +413,29 @@ class ExpansionCache { } +/** + * Read the cgroup memory limit once at startup. + * Returns the byte limit, or 0 if unavailable (disables the check). + */ +function readMemoryLimit() { + try { + const raw = require('fs').readFileSync('/sys/fs/cgroup/memory.max', 'utf8').trim(); + if (raw === 'max') return 0; // no cgroup limit + return parseInt(raw); + } catch { + return 0; // not on Linux / no cgroup + } +} + +const MEMORY_LIMIT = readMemoryLimit(); +const MEMORY_FRACTION = 0.98; +const MEMORY_THRESHOLD = MEMORY_LIMIT > 0 ? MEMORY_LIMIT * MEMORY_FRACTION : 0; // 90% of cgroup limit +const CHECK_FREQUENCY = 100; + class OperationContext { + // Shared counter across all instances — only check RSS every CHECK_FREQUENCY calls + static _checkCounter = 0; + constructor(langs, i18n = null, id = null, timeLimit = 30, resourceCache = null, expansionCache = null) { this.i18n = i18n; this.langs = this._ensureLanguages(langs); @@ -445,8 +467,8 @@ class OperationContext { */ copy() { const newContext = new OperationContext( - this.langs, this.i18n, this.id, this.timeLimit / 1000, - this.resourceCache, this.expansionCache + this.langs, this.i18n, this.id, this.timeLimit / 1000, + this.resourceCache, this.expansionCache ); newContext.contexts = [...this.contexts]; newContext.startTime = this.startTime; @@ -458,31 +480,64 @@ class OperationContext { } /** - * Check if operation has exceeded time limit + * Check if operation has exceeded time limit, or is pushing is over the memory limit * Skipped when running under debugger + * + * note: if the server pushes over the memory limit for the process, the process is terminated. + * the memory check here is intended to prevent process termination on the grounds that some + * big operation is pushing the limit. It might not be the big operation that is terminated first, + * but eventually it'll get terminated. + * + * this is called a *lot* so it's important to be efficient. Only check every CHECK_FREQUENCY + * times means that there could be a small overrun, but it's called often enough that the + * overrun won't be that signiifcant + * * @param {string} place - Location identifier for debugging * @returns {boolean} true if operation should be terminated */ deadCheck(place = 'unknown') { - // Skip time limit checks when debugging if (this.debugging) { return false; } - const elapsed = performance.now() - this.startTime; + OperationContext._checkCounter++; + if (OperationContext._checkCounter < CHECK_FREQUENCY) { + return false; + } + OperationContext._checkCounter = 0; + // Time check + const elapsed = performance.now() - this.startTime; if (elapsed > this.timeLimit) { const timeInSeconds = Math.round(this.timeLimit / 1000); this.log(`Operation took too long @ ${place} (${this.constructor.name})`); - - const error = new Issue("error", "too-costly", null, `Operation exceeded time limit of ${timeInSeconds} seconds at ${place}`); + const error = new Issue("error", "too-costly", null, + `Operation exceeded time limit of ${timeInSeconds} seconds at ${place}`); error.diagnostics = this.diagnostics(); throw error; } + // Memory check (piggyback on same sample) + if (MEMORY_THRESHOLD > 0) { + const rss = process.memoryUsage.rss(); + if (rss > MEMORY_THRESHOLD) { + const usedGB = (rss / 1024 / 1024 / 1024).toFixed(1); + const limitGB = (MEMORY_LIMIT / 1024 / 1024 / 1024).toFixed(1); + this.log(`Memory Limit: ${usedGB} GB of ${limitGB} GB limit @ ${place}`); + const error = new Issue("error", "too-costly", null, + `Operation aborted: server memory usage (${usedGB} GB) exceeds safe threshold (${MEMORY_FRACTION * 100}% of ${limitGB} GB limit) at ${place}`); + error.diagnostics = this.diagnostics(); + throw error; + } + } + return false; } + unSeeAll() { + this.contexts = []; + } + /** * Track a context URL and detect circular references * @param {string} vurl - Value set URL to track diff --git a/tx/tx-html.js b/tx/tx-html.js index 20d1b85d..be62b6f1 100644 --- a/tx/tx-html.js +++ b/tx/tx-html.js @@ -319,7 +319,7 @@ class TxHtmlRenderer { case 'Parameters': return await this.renderParameters(json); case 'CodeSystem': - return await this.renderCodeSystem(json, inBundle, _fmt, op); + return await this.renderCodeSystem(json, inBundle, _fmt, op, req.sourcePackage); case 'ValueSet': { let exp = undefined; if (!inBundle && !op && (!_fmt || _fmt == 'html')) { @@ -330,10 +330,10 @@ class TxHtmlRenderer { exp = error; } } - return await this.renderValueSet(json, inBundle, _fmt, op, exp); + return await this.renderValueSet(json, inBundle, _fmt, op, exp, req.sourcePackage); } case 'ConceptMap': - return await this.renderConceptMap(json, inBundle, _fmt, op); + return await this.renderConceptMap(json, inBundle, _fmt, op, req.sourcePackage); case 'CapabilityStatement': return await this.renderCapabilityStatement(json, inBundle); case 'TerminologyCapabilities': @@ -632,7 +632,7 @@ class TxHtmlRenderer { /** * Render CodeSystem resource */ - async renderCodeSystem(json, inBundle, _fmt) { + async renderCodeSystem(json, inBundle, _fmt, op, sourcePackage) { if (inBundle) { return await this.renderResourceWithNarrative(json, await this.renderer.renderCodeSystem(json)); } else { @@ -645,7 +645,7 @@ class TxHtmlRenderer { html += ``; if (!_fmt || _fmt == 'html') { - html += await this.renderResourceWithNarrative(json, await this.renderer.renderCodeSystem(json)); + html += await this.renderResourceWithNarrative(json, await this.renderer.renderCodeSystem(json, sourcePackage)); } else if (_fmt == "html/json") { html += await this.renderResourceJson(json); } else if (_fmt == "html/xml") { diff --git a/tx/tx.js b/tx/tx.js index 9b059512..5e605916 100644 --- a/tx/tx.js +++ b/tx/tx.js @@ -173,6 +173,7 @@ class TXModule { this.metadataHandler = new MetadataHandler({ baseUrl: config.baseUrl, serverVersion: packageJson.version, + txVersion: packageJson.txVersion, softwareName: config.softwareName || 'FHIRsmith', name: config.name || 'FHIRTerminologyServer', title: config.title || 'FHIR Terminology Server', diff --git a/tx/workers/expand.js b/tx/workers/expand.js index f5032c30..28671c12 100644 --- a/tx/workers/expand.js +++ b/tx/workers/expand.js @@ -207,6 +207,7 @@ class ValueSetExpander { reportedSupplements = new Set(); internalLimit = INTERNAL_DEFAULT_LIMIT; externalLimit = EXTERNAL_DEFAULT_LIMIT; + noDetails = false; constructor(worker, params) { this.worker = worker; @@ -336,7 +337,7 @@ class ValueSetExpander { } } - if (expansion) { + if (expansion && !this.noDetails) { const s = this.canonical(system, version); this.addParamUri(expansion, 'used-codesystem', s); if (cs != null) { @@ -362,113 +363,115 @@ class ValueSetExpander { if (isInactive) { n.inactive = true; } + if (!this.noDetails) { + if (status && status.toLowerCase() !== 'active') { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#status', 'status', "valueCode", status); + } else if (deprecated) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#status', 'status', "valueCode", 'deprecated'); + } - if (status && status.toLowerCase() !== 'active') { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#status', 'status', "valueCode", status); - } else if (deprecated) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#status', 'status', "valueCode", 'deprecated'); - } - - if (Extensions.has(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-label')) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#label', 'label', "valueString", Extensions.readString(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-label')); - } - if (Extensions.has(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-label')) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#label', 'label', "valueString", Extensions.readString(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-label')); - } + if (Extensions.has(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-label')) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#label', 'label', "valueString", Extensions.readString(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-label')); + } + if (Extensions.has(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-label')) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#label', 'label', "valueString", Extensions.readString(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-label')); + } - if (Extensions.has(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-conceptOrder')) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#order', 'order', "valueDecimal", Extensions.readNumber(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-conceptOrder', undefined)); - } - if (Extensions.has(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-conceptOrder')) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#order', 'order', "valueDecimal", Extensions.readNumber(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-conceptOrder', undefined)); - } + if (Extensions.has(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-conceptOrder')) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#order', 'order', "valueDecimal", Extensions.readNumber(csExtList, 'http://hl7.org/fhir/StructureDefinition/codesystem-conceptOrder', undefined)); + } + if (Extensions.has(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-conceptOrder')) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#order', 'order', "valueDecimal", Extensions.readNumber(vsExtList, 'http://hl7.org/fhir/StructureDefinition/valueset-conceptOrder', undefined)); + } - if (Extensions.has(csExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight')) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#itemWeight', 'weight', "valueDecimal", Extensions.readNumber(csExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight', undefined)); - } - if (Extensions.has(vsExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight')) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#itemWeight', 'weight', "valueDecimal", Extensions.readNumber(vsExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight', undefined)); - } + if (Extensions.has(csExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight')) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#itemWeight', 'weight', "valueDecimal", Extensions.readNumber(csExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight', undefined)); + } + if (Extensions.has(vsExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight')) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#itemWeight', 'weight', "valueDecimal", Extensions.readNumber(vsExtList, 'http://hl7.org/fhir/StructureDefinition/itemWeight', undefined)); + } - if (csExtList != null) { - for (const ext of csExtList) { - if (['http://hl7.org/fhir/StructureDefinition/coding-sctdescid', 'http://hl7.org/fhir/StructureDefinition/rendering-style', - 'http://hl7.org/fhir/StructureDefinition/rendering-xhtml', 'http://hl7.org/fhir/StructureDefinition/codesystem-alternate'].includes(ext.url)) { - if (!n.extension) {n.extension = []} - n.extension.push(ext); - } - if (['http://hl7.org/fhir/StructureDefinition/structuredefinition-standards-status'].includes(ext.url)) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#status', 'status', "valueCode", getValuePrimitive(ext)); + if (csExtList != null) { + for (const ext of csExtList) { + if (['http://hl7.org/fhir/StructureDefinition/coding-sctdescid', 'http://hl7.org/fhir/StructureDefinition/rendering-style', + 'http://hl7.org/fhir/StructureDefinition/rendering-xhtml', 'http://hl7.org/fhir/StructureDefinition/codesystem-alternate'].includes(ext.url)) { + if (!n.extension) { + n.extension = [] + } + n.extension.push(ext); + } + if (['http://hl7.org/fhir/StructureDefinition/structuredefinition-standards-status'].includes(ext.url)) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#status', 'status', "valueCode", getValuePrimitive(ext)); + } } } - } - if (vsExtList != null) { - for (const ext of vsExtList || []) { - if (['http://hl7.org/fhir/StructureDefinition/valueset-supplement', 'http://hl7.org/fhir/StructureDefinition/valueset-deprecated', - 'http://hl7.org/fhir/StructureDefinition/structuredefinition-standards-status', - 'http://hl7.org/fhir/StructureDefinition/valueset-concept-definition', 'http://hl7.org/fhir/StructureDefinition/coding-sctdescid', - 'http://hl7.org/fhir/StructureDefinition/rendering-style', 'http://hl7.org/fhir/StructureDefinition/rendering-xhtml'].includes(ext.url)) { - if (!n.extension) {n.extension = []} - n.extension.push(ext); + if (vsExtList != null) { + for (const ext of vsExtList || []) { + if (['http://hl7.org/fhir/StructureDefinition/valueset-supplement', 'http://hl7.org/fhir/StructureDefinition/valueset-deprecated', + 'http://hl7.org/fhir/StructureDefinition/structuredefinition-standards-status', + 'http://hl7.org/fhir/StructureDefinition/valueset-concept-definition', 'http://hl7.org/fhir/StructureDefinition/coding-sctdescid', + 'http://hl7.org/fhir/StructureDefinition/rendering-style', 'http://hl7.org/fhir/StructureDefinition/rendering-xhtml'].includes(ext.url)) { + if (!n.extension) { + n.extension = [] + } + n.extension.push(ext); + } } } - } - // display and designations - const pref = displays.preferredDesignation(this.params.workingLanguages(), this.reportedSupplements); - if (pref && pref.value) { - n.display = pref.value; - } + // display and designations + const pref = displays.preferredDesignation(this.params.workingLanguages(), this.reportedSupplements); + if (pref && pref.value) { + n.display = pref.value; + } - if (this.params.includeDesignations) { - for (const t of displays.designations) { - if (t !== pref && this.useDesignation(t) && t.value != null && !this.redundantDisplay(n, t.language, t.use, t.value)) { - if (!n.designation) { - n.designation = []; - } - if (t.source) { - this.reportedSupplements.add(t.source); + if (this.params.includeDesignations) { + for (const t of displays.designations) { + if (t !== pref && this.useDesignation(t) && t.value != null && !this.redundantDisplay(n, t.language, t.use, t.value)) { + if (!n.designation) { + n.designation = []; + } + if (t.source) { + this.reportedSupplements.add(t.source); + } + n.designation.push(t.asObject()); } - n.designation.push(t.asObject()); } } - } - for (const pn of this.params.properties) { - if (pn === 'definition') { - if (definition) { - this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#definition', pn, "valueString", definition); - } - } else if (pn === 'usage-count') { - let counter = cs.usages().get(code); - this.defineProperty(expansion, n, 'http://fhir.org/FHIRsmith/CodeSystem/concept-properties#usage-count', pn, "valueInteger", counter ? counter.count : 0); - } else if (csProps != null && cs != null) { - for (const cp of csProps) { - if (cp.code === pn) { - let vn = getValueName(cp); - let v = cp[vn]; - this.defineProperty(expansion, n, this.getPropUrl(cs, pn, cp), pn, vn, v); + for (const pn of this.params.properties) { + if (pn === 'definition') { + if (definition) { + this.defineProperty(expansion, n, 'http://hl7.org/fhir/concept-properties#definition', pn, "valueString", definition); + } + } else if (pn === 'usage-count') { + let counter = cs.usages().get(code); + this.defineProperty(expansion, n, 'http://fhir.org/FHIRsmith/CodeSystem/concept-properties#usage-count', pn, "valueInteger", counter ? counter.count : 0); + } else if (csProps != null && cs != null) { + for (const cp of csProps) { + if (cp.code === pn) { + let vn = getValueName(cp); + let v = cp[vn]; + this.defineProperty(expansion, n, this.getPropUrl(cs, pn, cp), pn, vn, v); + } } } } } - - if (!this.map.has(s)) { - this.fullList.push(n); - this.map.set(s, n); - if (parent != null) { - if (!parent.contains) { - parent.contains = []; - } - parent.contains.push(n); - } else { - this.rootList.push(n); + this.fullList.push(n); + this.map.set(s, n); + if (parent != null && !this.noDetails) { + if (!parent.contains) { + parent.contains = []; } + parent.contains.push(n); } else { - this.canBeHierarchy = false; + this.rootList.push(n); } result = n; + } else { + this.canBeHierarchy = false; } return result; } @@ -777,9 +780,17 @@ class ValueSetExpander { const c = await cs.filterConcept(ctxt, set[0]); if (await this.passesFilters(cs, c, prep, set, 1)) { const cds = new Designations(this.worker.i18n.languageDefinitions); - await this.listDisplaysFromProvider(cds, cs, c); - let added = await this.includeCode(cs, null, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), await cs.isDeprecated(c), await cs.getStatus(c), - cds, await cs.definition(c), await cs.itemWeight(c), expansion, valueSets, await cs.extensions(c), null, await cs.properties(c), null, excludeInactive, vsSrc.url); + let added; + if (this.noDetails) { + await this.listDisplaysFromProvider(cds, cs, c); + added = await this.includeCode(cs, null, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), false, null, + cds, null, null, expansion, valueSets, null, null, null, null, excludeInactive, vsSrc.url); + + } else { + await this.listDisplaysFromProvider(cds, cs, c); + added = await this.includeCode(cs, null, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), await cs.isDeprecated(c), await cs.getStatus(c), + cds, await cs.definition(c), await cs.itemWeight(c), expansion, valueSets, await cs.extensions(c), null, await cs.properties(c), null, excludeInactive, vsSrc.url); + } if (added) { this.addToTotal(); } @@ -799,15 +810,21 @@ class ValueSetExpander { Extensions.checkNoModifiers(cc, 'ValueSetExpander.processCodes', 'set concept reference', vsSrc.vurl); const cctxt = await cs.locate(cc.code, this.allAltCodes); if (cctxt && cctxt.context && (!this.params.activeOnly || !await cs.isInactive(cctxt.context)) && await this.passesFilters(cs, cctxt.context, prep, filters, 0)) { - await this.listDisplaysFromProvider(cds, cs, cctxt.context); - this.listDisplaysFromIncludeConcept(cds, cc, vsSrc); - if (filter.passesDesignations(cds) || filter.passes(cc.code)) { - let ov = Extensions.readString(cc, 'http://hl7.org/fhir/StructureDefinition/itemWeight'); - if (!ov) { - ov = await cs.itemWeight(cctxt.context); + let added; + if (this.noDetails) { + added = await this.includeCode(cs, null, cs.system(), cs.version(), cc.code, await cs.isAbstract(cctxt.context), await cs.isInactive(cctxt.context), null, null, cds, + null, null, expansion, valueSets, null, null, null, null, excludeInactive, vsSrc.url); + } else { + await this.listDisplaysFromProvider(cds, cs, cctxt.context); + this.listDisplaysFromIncludeConcept(cds, cc, vsSrc); + if (filter.passesDesignations(cds) || filter.passes(cc.code)) { + let ov = Extensions.readString(cc, 'http://hl7.org/fhir/StructureDefinition/itemWeight'); + if (!ov) { + ov = await cs.itemWeight(cctxt.context); + } + added = await this.includeCode(cs, null, cs.system(), cs.version(), cc.code, await cs.isAbstract(cctxt.context), await cs.isInactive(cctxt.context), await cs.isDeprecated(cctxt.context), await cs.getStatus(cctxt.context), cds, + await cs.definition(cctxt.context), ov, expansion, valueSets, await cs.extensions(cctxt.context), cc.extension, await cs.properties(cctxt.context), null, excludeInactive, vsSrc.url); } - let added = await this.includeCode(cs, null, cs.system(), cs.version(), cc.code, await cs.isAbstract(cctxt.context), await cs.isInactive(cctxt.context), await cs.isDeprecated(cctxt.context), await cs.getStatus(cctxt.context), cds, - await cs.definition(cctxt.context), ov, expansion, valueSets, await cs.extensions(cctxt.context), cc.extension, await cs.properties(cctxt.context), null, excludeInactive, vsSrc.url); if (added) { this.addToTotal(); } @@ -850,24 +867,33 @@ class ValueSetExpander { } this.worker.opContext.log('iterate filters'); + const cds = new Designations(this.worker.i18n.languageDefinitions); while (await cs.filterMore(prep, fset[0])) { this.worker.deadCheck('processCodes#5'); const c = await cs.filterConcept(prep, fset[0]); const ok = (!this.params.activeOnly || !await cs.isInactive(c)) && (await this.passesFilters(cs, c, prep, fset, 1)); if (ok) { + cds.clear(); // count++; - const cds = new Designations(this.worker.i18n.languageDefinitions); if (this.passesImports(valueSets, cs.system(), await cs.code(c), 0)) { - await this.listDisplaysFromProvider(cds, cs, c); - let parent = null; - if (cs.hasParents()) { - parent = this.map.get(this.keyS(cs.system(), cs.version(), await cs.parent(c))); + let added; + if (this.noDetails) { + added = await this.includeCode(cs, null, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), + null, null, cds, null, null, + expansion, null, null, null, null, null, excludeInactive, vsSrc.url); + } else { - this.canBeHierarchy = false; + await this.listDisplaysFromProvider(cds, cs, c); + let parent = null; + if (cs.hasParents()) { + parent = this.map.get(this.keyS(cs.system(), cs.version(), await cs.parent(c))); + } else { + this.canBeHierarchy = false; + } + added = await this.includeCode(cs, parent, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), + await cs.isDeprecated(c), await cs.getStatus(c), cds, await cs.definition(c), await cs.itemWeight(c), + expansion, null, await cs.extensions(c), null, await cs.properties(c), null, excludeInactive, vsSrc.url); } - let added = await this.includeCode(cs, parent, await cs.system(), await cs.version(), await cs.code(c), await cs.isAbstract(c), await cs.isInactive(c), - await cs.isDeprecated(c), await cs.getStatus(c), cds, await cs.definition(c), await cs.itemWeight(c), - expansion, null, await cs.extensions(c), null, await cs.properties(c), null, excludeInactive, vsSrc.url); if (added) { this.addToTotal(); } @@ -1053,9 +1079,16 @@ class ValueSetExpander { let n = null; if ((!this.params.excludeNotForUI || !await cs.isAbstract(context)) && (!this.params.activeOnly || !await cs.isInactive(context))) { const cds = new Designations(this.worker.i18n.languageDefinitions); - await this.listDisplaysFromProvider(cds, cs, context); - const t = await this.includeCode(cs, parent, await cs.system(), await cs.version(), context.code, await cs.isAbstract(context), await cs.isInactive(context), await cs.isDeprecated(context), await cs.getStatus(context), cds, await cs.definition(context), - await cs.itemWeight(context), expansion, imports, await cs.extensions(context), null, await cs.properties(context), null, excludeInactive, srcUrl); + let t; + if (this.noDetails) { + t = await this.includeCode(cs, null, await cs.system(), await cs.version(), context.code, await cs.isAbstract(context), await cs.isInactive(context), null, null, null, + null, expansion, imports, null, null, null, null, excludeInactive, srcUrl); + + } else { + await this.listDisplaysFromProvider(cds, cs, context); + t = await this.includeCode(cs, parent, await cs.system(), await cs.version(), context.code, await cs.isAbstract(context), await cs.isInactive(context), await cs.isDeprecated(context), await cs.getStatus(context), cds, await cs.definition(context), + await cs.itemWeight(context), expansion, imports, await cs.extensions(context), null, await cs.properties(context), null, excludeInactive, srcUrl); + } if (t != null) { result++; } @@ -1189,7 +1222,7 @@ class ValueSetExpander { return result; // just return the expansion } - if (this.params.generateNarrative) { + if (this.params.generateNarrative && !this.noDetails) { div_ = div(); table = div_.table("grid"); } else { @@ -1203,8 +1236,10 @@ class ValueSetExpander { if (this.params.limit <= 0) { this.limitCount = this.externalLimit; - } else { + } else if (this.externalLimit) { this.limitCount = Math.min(this.params.limit, this.externalLimit); + } else { + this.limitCount = this.params.limit; } this.offset = this.params.offset; this.count = this.params.count; @@ -1321,8 +1356,10 @@ class ValueSetExpander { exp.total = this.total; } list = this.fullList; - for (const c of this.fullList) { - c.contains = undefined; + if (!this.noDetails) { + for (const c of this.fullList) { + c.contains = undefined; + } } if (table != null) { div_.addTag('p').setAttribute('style', 'color: Navy').tx('Because of the way that this value set is defined, not all the possible codes can be listed in advance'); diff --git a/tx/workers/metadata.js b/tx/workers/metadata.js index 2eebf4da..758b3eac 100644 --- a/tx/workers/metadata.js +++ b/tx/workers/metadata.js @@ -107,10 +107,11 @@ class MetadataHandler { * @returns {Object} CapabilityStatement resource */ buildCapabilityStatement(endpoint) { + const now = new Date().toISOString(); const fhirVersion = this.mapFhirVersion(endpoint.fhirVersion); const baseUrl = this.config.baseUrl || `https://${this.host}${endpoint.path}`; - const serverVersion = this.config.serverVersion || '1.0.0'; + const serverVersion = this.config.serverVersion; return { resourceType: 'CapabilityStatement', @@ -124,7 +125,7 @@ class MetadataHandler { }, { 'url' : 'value', - 'valueCode' : '1.8.0' + 'valueCode' : this.config.txVersion }, { 'extension' : [ diff --git a/tx/workers/read.js b/tx/workers/read.js index 85ac90d0..e0fd04e0 100644 --- a/tx/workers/read.js +++ b/tx/workers/read.js @@ -80,6 +80,7 @@ class ReadWorker extends TerminologyWorker { async handleCodeSystem(req, res, id) { let cs = this.provider.getCodeSystemById(this.opContext, id); if (cs != null) { + req.sourcePackage = cs.sourcePackage; return res.json(cs.jsonObj); } @@ -145,6 +146,7 @@ class ReadWorker extends TerminologyWorker { this.deadCheck('handleValueSet-loop'); const vs = await vsp.fetchValueSetById(id); if (vs) { + req.sourcePackage = vs.sourcePackage; return res.json(vs.jsonObj); } } @@ -165,9 +167,10 @@ class ReadWorker extends TerminologyWorker { // Iterate through valueSetProviders in order for (const cmsp of this.provider.conceptMapProviders) { this.deadCheck('handleConceptMap-loop'); - const vs = await cmsp.fetchConceptMapById(id); - if (vs) { - return res.json(vs.jsonObj); + const cm = await cmsp.fetchConceptMapById(id); + if (cm) { + req.sourcePackage = cm.sourcePackage; + return res.json(cm.jsonObj); } } diff --git a/tx/workers/related.js b/tx/workers/related.js index 55b5e382..acb698b3 100644 --- a/tx/workers/related.js +++ b/tx/workers/related.js @@ -18,8 +18,9 @@ const {SearchFilterText} = require("../library/designations"); const {ArrayMatcher} = require("../../library/utilities"); const {debugLog} = require("../operation-context"); - class RelatedWorker extends TerminologyWorker { + showLogic = false; + /** * @param {OperationContext} opContext - Operation context * @param {Logger} log - Logger instance @@ -115,7 +116,6 @@ class RelatedWorker extends TerminologyWorker { this.setupAdditionalResources(params); let txp = new TxParameters(this.opContext.i18n.languageDefinitions, this.opContext.i18n, false); txp.readParams(params); - this.params = txp; let thisVS = await this.readValueSet(res, "this", params, txp); @@ -199,6 +199,7 @@ class RelatedWorker extends TerminologyWorker { } async doRelated(txp, thisVS, otherVS) { + // ok, we have to compare the composes. we don't care about anything else const thisC = thisVS.jsonObj.compose; const otherC = otherVS.jsonObj.compose; @@ -213,28 +214,35 @@ class RelatedWorker extends TerminologyWorker { Extensions.checkNoModifiers(otherC, 'RelatedWorker.doRelated', 'compose', otherVS.vurl) this.checkNoLockedDate(otherVS.vurl, otherC); - let systems = new Map(); // tracks whether they are version dependent or not + let systems = new Map(); // tracks whether the comparison is version dependent or not // ok, first, if we can determine that the value sets match from the definitions, we will // if that fails, then we have to do the expansions, and then decide + let allCriteria = [...thisC.include || [], ...thisC.exclude || [], ...otherC.include || [], ...otherC.exclude || []]; // first, we sort the includes by system, and then compare them as a group // Build a map of system -> { this: [...includes], other: [...includes] } const systemMap = new Map(); - await this.addIncludes(systems, systemMap, thisC.include || [], 'this', txp); - await this.addIncludes(systems, systemMap, otherC.include || [], 'other', txp); - await this.addIncludes(systems, systemMap, thisC.exclude || [], 'thisEx', txp); - await this.addIncludes(systems, systemMap, otherC.exclude || [], 'otherEx', txp); - - let status = { left: false, right: false, fail: false, common : false}; - - for (const [key, value] of systemMap.entries()) { - if (key) { - let cs = await this.findCodeSystem(key, null, txp, ['complete', 'fragment'], null, true); - await this.compareSystems(systems, status, cs, value); - } else { - this.compareNonSystems(status, value); + await this.addIncludes(systems, systemMap, thisC.include || [], 'this', txp, allCriteria); + await this.addIncludes(systems, systemMap, otherC.include || [], 'other', txp, allCriteria); + await this.addIncludes(systems, systemMap, thisC.exclude || [], 'thisEx', txp, allCriteria); + await this.addIncludes(systems, systemMap, otherC.exclude || [], 'otherEx', txp, allCriteria); + + let status = { empty: false, left: false, right: false, fail: false, common : false}; + let diagnostics = {}; + + let canBeQuick = !this.hasMultipleVersionsForAnySystem(systems, systemMap); + if (canBeQuick) { + for (const [key, value] of systemMap.entries()) { + if (key) { + let cs = await this.findCodeSystem(key, null, txp, ['complete', 'fragment'], null, true); + await this.compareSystems(systems, status, cs, value, diagnostics); + } else { + this.compareNonSystems(status, value, diagnostics); + } } + } else { + status.fail = true; } let exp = false; @@ -242,13 +250,15 @@ class RelatedWorker extends TerminologyWorker { // expansions might not work (infinite value sets) so // we can't tell. if (status.fail) { - status.fail = false; + status = { left: false, right: false, fail: false, common : false}; // reset; exp = true; - await this.compareExpansions(systems, status, thisVS, otherVS); + await this.compareExpansions(systems, status, thisVS, otherVS, diagnostics); } let outcome; if (status.fail) { outcome = this.makeOutcome("indeterminate", `Unable to compare ${thisVS.vurl} and ${otherVS.vurl}: `+status.reason); + } else if (status.empty) { + outcome = this.makeOutcome("empty", `Both the value sets ${thisVS.vurl} and ${otherVS.vurl} are empty`); } else if (!status.common) { outcome = this.makeOutcome("disjoint", `No shared codes between the value sets ${thisVS.vurl} and ${otherVS.vurl}`); } else if (!status.left && !status.right) { @@ -260,36 +270,56 @@ class RelatedWorker extends TerminologyWorker { } else { outcome = this.makeOutcome("subset", `The valueSet ${thisVS.vurl} is a seb-set of the valueSet ${otherVS.vurl}`); } - if (exp) { - outcome.parameter.push({name: 'expansion', valueBoolean: exp}) + if (txp.diagnostics) { + outcome.parameter.push({name: 'performed-expansion', valueBoolean: exp ? true : false}) + if (diagnostics.missing && diagnostics.missing.length > 0) { + outcome.parameter.push({name: 'missing-codes', valueString: diagnostics.missing.map(c => c.code).join(',') }) + } + if (diagnostics.extra && diagnostics.extra.length > 0) { + outcome.parameter.push({name: 'extra-codes', valueString: diagnostics.extra.map(c => c.code).join(',') }) + } + if (diagnostics.common && diagnostics.common.length > 0) { + outcome.parameter.push({name: 'common-codes', valueString: diagnostics.common.map(c => c.left.code).join(',') }) + } + if (!exp) { + if (diagnostics.missingCodes && diagnostics.missingCodes.length > 0) { + outcome.parameter.push({name: 'missing-codes', valueString: diagnostics.missingCodes.join(',')}) + } + if (diagnostics.extraCodes && diagnostics.extraCodes.length > 0) { + outcome.parameter.push({name: 'extra-codes', valueString: diagnostics.extraCodes.join(',')}) + } + if (diagnostics.commonCodes && diagnostics.commonCodes.length > 0) { + outcome.parameter.push({name: 'common-codes', valueString: diagnostics.commonCodes.join(',')}) + } + } } return outcome; } - async addIncludes(systems, systemMap, includes, side, txp) { + async addIncludes(systems, systemMap, includes, side, txp, allCriteria) { for (const inc of includes) { let key = inc.system || ''; let v = {}; - if (await this.versionMatters(systems, key, inc.version, v, txp)) { + if (await this.versionMatters(systems, key, inc.version, v, txp, allCriteria)) { key = key + "|" + v.version; } if (!systemMap.has(key)) { - systemMap.set(key, {this: [], other: []}); + systemMap.set(key, {this: [], other: [], thisEx: [], otherEx: []}); } systemMap.get(key)[side].push(inc); } } - async versionMatters(systems, key, version, v, txp) { - if (systems.has(key)) { - return systems.get(key); - } + async versionMatters(systems, key, version, v, txp, allCriteria) { let cs = await this.findCodeSystem(key, version, txp, ['complete', 'fragment'], null, true); - let res = cs == null || cs.versionNeeded(); + let alreadyVersionDependent = systems.has(key) && systems.get(key).criteria; + let res = cs != null && (alreadyVersionDependent || ((version || cs.version()) && (cs.versionNeeded() || this.anyCriteriaHasFilters(allCriteria, key)))); // if there's filters, the version always matters if (res) { v.version = version || cs ? cs.version() : undefined; } - systems.set(key, res); + if (!systems.has(key)) { + systems.set(key, {criteria: res, codes: cs ? cs.versionNeeded() : false}); + } return res; } @@ -298,8 +328,8 @@ class RelatedWorker extends TerminologyWorker { status.fail = true; } - async compareSystems(systems, status, cs, value) { - if (value.thisEx || value.otherEx) { + async compareSystems(systems, status, cs, value, diagnostics) { + if ((value.thisEx && value.thisEx.length > 0) || (value.otherEx && value.otherEx.length > 0)) { // we don't try in this case status.fail = true; status.common = true; @@ -341,25 +371,33 @@ class RelatedWorker extends TerminologyWorker { status.common = true; status.right = true; return; - } else if (this.isConcepts(value.this[0]) && this.isConcepts(value.other[0])) { - this.compareCodeLists(status, value.this[0], value.other[0]); - return; - } else if (this.isFilter(value.this[0]) && this.isFilter(value.other[0])) { + } else if (value.this.length > 1 || value.other.length > 1) { + status.common = true; + // if we have mixed concepts, or multiple filters, we can't reason about them (too many scenarios where they overlap in + // unpredictable ways. If they're not identical, we fail if (value.this.length != value.other.length) { status.fail = true; - return; } else { for (let i = 0; i < value.this.length; i++) { let t = value.this[i]; let o = value.other[i]; - if (!await this.filterSetsMatch(status, cs, t, o)) { + if (!this.includesIdentical(t, o)) { status.fail = true; - return; + break; } - status.common = true; - return; } } + return; + } else if (this.isConcepts(value.this[0]) && this.isConcepts(value.other[0])) { + this.compareCodeLists(status, value.this[0], value.other[0], diagnostics); + return; + } else if (this.isFilter(value.this[0]) && this.isFilter(value.other[0])) { + let t = value.this[0]; + let o = value.other[0]; + if (!await this.filterSetsMatch(status, cs, t, o)) { + status.fail = true; + } + return; } } status.fail = true; // not sure why we got to here, but it doesn't matter: we can't tell @@ -383,6 +421,15 @@ class RelatedWorker extends TerminologyWorker { return false; } + hasFilters(list) { + for (const inc of list) { + if (inc.filter?.length > 0) { + return true; + } + } + return false; + } + tidyIncludes(list) { let collector = null; for (let i = list.length - 1; i >= 0; i--) { @@ -431,13 +478,16 @@ class RelatedWorker extends TerminologyWorker { ); } - compareCodeLists(status, t, o) { + compareCodeLists(status, t, o, diagnostics) { const tSet = new Set(t.concept.map(x => x.code)); const oSet = new Set(o.concept.map(x => x.code)); - status.common = [...tSet].filter(c => oSet.has(c)).length > 0; - status.left = [...tSet].filter(c => !oSet.has(c)).length > 0; - status.right = [...oSet].filter(c => !tSet.has(c)).length > 0; + diagnostics.commonCodes = [...tSet].filter(c => oSet.has(c)); + diagnostics.missingCodes = [...tSet].filter(c => !oSet.has(c)); + diagnostics.extraCodes = [...oSet].filter(c => !tSet.has(c)); + status.common = diagnostics.commonCodes.length > 0; + status.left = diagnostics.missingCodes.length > 0; + status.right =diagnostics.extraCodes.length > 0; } makeOutcome(code, msg) { @@ -457,15 +507,45 @@ class RelatedWorker extends TerminologyWorker { return !inc.concept && !inc.filter; } - async compareExpansions(systems, status, thisC, otherC) { - const expThis = await this.doExpand(thisC); - const expOther = await this.doExpand(otherC); + async compareExpansions(systems, status, thisC, otherC, diagnostics) { + + const expResThis = await this.doExpand(thisC); + this.opContext.unSeeAll(); + const expResOther = await this.doExpand(otherC); + if (expResThis.error || expResOther.error) { + status.fail = true; + if (expResThis.error && expResOther.error) { + if (expResThis.error == expResOther.error) { + status.reason = "Both expansions failed: "+expResThis.error.message; + } else { + status.reason = "Both expansions failed with different errors: "+expResThis.error.message+"; "+expResOther.error.message; + } + } else if (expResThis.error) { + status.reason = "This expansion failed: "+expResThis.error.message + } else { + status.reason = "Other expansion failed: "+expResOther.error.message + } + return; + } + let expThis = expResThis.vs; + let expOther = expResOther.vs; if (this.isUnclosed(expThis) || this.isUnclosed(expOther)) { status.fail = true; + if (this.isUnclosed(expThis) && this.isUnclosed(expOther)) { + status.reason = "Both expansions are unclosed." + } else if (this.isUnclosed(expThis)) { + status.reason = "This expansion is unclosed." + } else { + status.reason = "Other expansion is unclosed." + } return; } + if ((!expThis.expansion.contains || expThis.expansion.contains.length == 0) && (!expOther.expansion.contains || expOther.expansion.contains.length == 0)) { + status.empty = true; + return; + } const matcher = new ArrayMatcher((l, r) => this.matchContains(systems, l, r) ); @@ -482,6 +562,11 @@ class RelatedWorker extends TerminologyWorker { if (matcher.unmatchedRight.length > 0) { status.right = true; } + if (matcher.unmatchedLeft.length > 0 || matcher.unmatchedRight.length > 0) { + diagnostics.common = matcher.matched; + } + diagnostics.missing = matcher.unmatchedLeft; + diagnostics.extra = matcher.unmatchedRight; } isUnclosed(vs) { @@ -495,7 +580,7 @@ class RelatedWorker extends TerminologyWorker { if (thisC.code != otherC.code) { return false; } - let versionMatters = systems.get(thisC.system); + let versionMatters = systems.has(thisC.system) && systems.get(thisC.system).codes; if (versionMatters && thisC.version != otherC.version) { return false; } else { @@ -504,16 +589,25 @@ class RelatedWorker extends TerminologyWorker { } async doExpand(vs) { - let txpe = this.params.clone(); - txpe.limit = 10000; - txpe.excludeNested = true; - let exp = new ValueSetExpander(this, txpe); - let vse = await exp.expand(vs, new SearchFilterText(''), true); - return vse + try { + let txpe = this.params.clone(); + txpe.limit = 10000; + txpe.excludeNested = true; + let start = new Date(); + console.log("Expanding value set"); + let exp = new ValueSetExpander(this, txpe); + exp.noDetails = true; + let vse = await exp.expand(vs, new SearchFilterText(''), true); + console.log("Expanded value set - took " + (new Date() - start) + "ms"); + return {vs: vse, error: null}; + } catch (error) { + debugLog(error, "Error expanding value set"); + return {vs: null, error: error}; + } } isConcepts(inc) { - return inc.concept && inc.concept.length > 0; + return inc.concept && inc.concept.length > 0 && !this.isFilter(inc); } isFilter(inc) { @@ -522,52 +616,99 @@ class RelatedWorker extends TerminologyWorker { async filterSetsMatch(status, cs, t, o) { // two includes have matching filters if the set of filters match. - - let localstatus = { left: false, right: false}; - - const matcher = new ArrayMatcher((l, r) => - this.filtersMatch(localstatus, cs, l, r) - ); - await matcher.match(t.filter, o.filter); - - if (matcher.unmatchedLeft.length > 0 || matcher.unmatchedRight.length > 0) { + if (t.filter.length != o.filter.length) { return false; + } + if (t.filter.length > 1) { + t.filter.sort((a, b) => (a.property || '').localeCompare(b.property) || (a.op || '').localeCompare(b.op) || (a.value || '').localeCompare(b.value)); + o.filter.sort((a, b) => (a.property || '').localeCompare(b.property) || (a.op || '').localeCompare(b.op) || (a.value || '').localeCompare(b.value)) + // we can't draw any conclusions if there's more than one filter, and they aren't identical, + // because we don't guess how they might interact with each other + for (let i = 0; i < (t.filter || []).length; i++) { + if (t.filter[i].property !== o.filter[i].property || t.filter[i].op !== o.filter[i].op || t.filter[i].value !== o.filter[i].value) { + return false; + } + } + status.common = true; + return true; } else { - if (localstatus.left) { - status.left = true; + let tf = t.filter[0]; + let of = o.filter[0]; + if (tf.property != of.property || tf.op != of.op) { + return false; } - if (localstatus.right) { - status.right = true; + if (tf.value == of.value) { + status.common = true; + return true; + } else if (tf.op == 'is-a') { + let rel = await cs.subsumesTest(tf.value, of.value) + switch (rel) { + case 'equivalent': + return true; + case 'subsumes': + status.common = true; + status.left = true; + return true; + case 'subsumed-by': + status.common = true; + status.right = true; + return true; + default: + // we know that the codes aren't related, but we don't know whether they have common children + // well, that depends on whether there's a multi-heirarchy in play + if (!cs.hasMultiHierarchy()) { + status.common = false; + status.left = true; + status.right = true; + return true; + + } else { + return false; + } + } + } else { + return false; } - return true; } } - async filtersMatch(status, cs, t, o) { - if (t.property != o.property || t.op != o.op) { + includesIdentical(t, o) { + if ((t.concept || []).length !== (o.concept || []).length) { return false; } - if (t.value == o.value) { - return true; + for (let i = 0; i < (t.concept || []).length; i++) { + if (t.concept[i].code !== o.concept[i].code) { + return false; + } } - if (t.op == 'is-a') { - let rel = await cs.subsumesTest(t.value, o.value) - switch (rel) { - case 'equivalent': - return true; - case 'subsumes': - status.left = true; - return true; - case 'subsumed-by': - status.right = true; - return true; - default: - return false; + if ((t.filter || []).length !== (o.filter || []).length) { + return false; + } + for (let i = 0; i < (t.filter || []).length; i++) { + if (t.filter[i].property !== o.filter[i].property || t.filter[i].op !== o.filter[i].op || t.filter[i].value !== o.filter[i].value ) { + return false; } } - return false; + + return true; } + anyCriteriaHasFilters(allCriteria, key) { + return allCriteria.some(c => c.system === key && c.filter && c.filter.length > 0); + } + + hasMultipleVersionsForAnySystem(systems, systemMap) { + return [...systems.entries()].some(([url, val]) => { + if (val.criteria !== true) return false; + let count = 0; + for (const k of systemMap.keys()) { + if (k.startsWith(url)) { + count++; + } + } + return count > 1; + }); + } } module.exports = { diff --git a/xig/xig-template.html b/xig/xig-template.html index 28ea8cb4..97232330 100644 --- a/xig/xig-template.html +++ b/xig/xig-template.html @@ -90,7 +90,9 @@

FHIR © HL7.org 2011+.  |  FHIRsmith [%ver%] © HealthIntersections.com.au 2023+  |  - XIG built as of [%download-date%]  |  [%total-resources%] resources in [%total-packages%] packages  |  ([%ms%] ms) + XIG built as of [%download-date%]  |  [%total-resources%] resources in [%total-packages%] packages  | +   ([%ms%] ms) + [%sponsorMessage%]