Developer guide

Welcome to the developer guide of SMARTSexplore!

Installation (development mode)

Please refer to the user guide’s installation instructions, with the following differences:

  • Use python setup.py develop rather than python setup.py install. develop will let all imports like smartsexplore.* refer to the code directory and include all latest changes, while install will not let imports use any code changes until you run it again.

  • When developing frontend code, use npm run watch rather than npm run build. Doing so will keep a process running that will automatically rebuild the frontend code package, whenever changes are made in any frontend code (smartsexplore/frontend directory).

Management commands

There are a number of useful commands included with SMARTSexplore, for:

  • installing and building the software,

  • managing application data,

  • executing tests,

  • and generating documentation.

These are detailed in the following.

Managing the database with Flask commands

With flask db init, you can initialize the database with table definitions and base data. This is a required action before the application can run. It is also required before executing any of the SMARTS management commands.

For managing and generating SMARTS data, use flask smarts. Below is a listing of all available subcommands.

  • flask smarts: Manages SMARTS data.

    • flask smarts add_library: Add a single SMARTS library to the database.

    • flask smarts add_libraries: Add multiple SMARTS libraries to the database, named based on the the filenames of the given .smarts files.

    • flask smarts calculate_edges: Run SMARTScompare to calculate and store edges in the database. Can generate and store both directed and undirected edges. Note that undirected edges are currently not used by the frontend.

    • flask smarts draw_all_smarts: Draws SVG images of all SMARTS based on the SMARTSview visual language, and stores them in the Flask application’s instance folder.

    • flask smarts draw_all_subsets: Draws SVG images of all directed edges (subsets) that are stored in the database (should have been previously generated by flask smarts calculate_edges). Stores these images in the Flask application’s instance folder.

Python setup and documentation generation with setup.py

With python setup.py, you can use all Python package management features available from Python setuptools:

  • installing the software and its dependencies (python setup.py install / develop)

  • creating a distributable package (python setup.py sdist)

  • generating documentation (python setup.py build_sphinx)

Refer to python setup.py --help-commands to get a comprehensive list and explanations of all available commands.

Building the frontend code and keeping it up to date

SMARTSexplore uses the Webpack bundler to bundle all frontend code and assets together. The frontend application then includes this code bundle as one JavaScript file.

We use webpack via npm commands. There are two main modes for using Webpack, which are offered by the following commands:

  • npm run watch: Builds the frontend code bundle once in development mode, and then watches for changes in the frontend code files. Rebuilds whenever changes are detected. Highly recommended for frontend development.

  • npm run build: Builds the frontend code bundle once in productino mode, then exits. Highly recommended for production use.

Both commands generate and overwrite files in smartsexplore/static, in particular bundle.js.

Testing the application

There are three kinds of tests included in the application:

  • Frontend tests: npm run test_frontend

  • Backend tests: npm run test_backend

  • End-to-end tests: npm run test_e2e

Note

End-to-end tests require a Chrome browser and corresponding Selenium Webdriver to be installed. This is not managed by any available command and you’ll need to set it up yourself on your system.

To run all kinds of tests at once, you can use npm run test.

Generating test coverage

You can generate coverage statistics for the frontend and the backend. End-to-end test coverage of frontend code is included in the overall frontend coverage statistics.

To generate frontend coverage, use npm run coverage_frontend; for backend coverage, use npm run coverage_backend. Both commands will print a short overview to the console, and also write detailed browsable HTML output to the coverage/ directory.

Coverage implementation details

The coverage generation for the backend code is based on the pytest-cov package, without any additional configuration.

For the frontend coverage, we manually combine results from jest and from the E2E tests in test/end-to-end. The E2E tests retrieve coverage results from the automated browser and store them next to the jest coverage results (all JSON files). The browser coverage results are obtained by instrumenting the code with IstanbulJS, which is set up in the babel-loader configuration in the file webpack.e2e-test.js. (Jest also uses IstanbulJS internally.) We then run the Istanbul coverage report generator nyc on the combined coverage data, which generates an overall report on the console and as HTML.

Note

Running the end-to-end tests will overwrite the Webpack bundle (also when generating frontend coverage). Please run npm run build or npm run watch (whichever is appropriate for you) again after running these tests, if you want to use, distribute or deploy the software.

Code directory structure

Since there are a lot of files and directories in the code repository, the following list explains the purpose of each of them.

  • babel.config.js: Configuration for the Babel JavaScript build tool, used by Webpack for all builds. Transforms ES6 Modules to CommonJS modules, to ensure cross-browser and Node compatibility. Essential to make end-to-end tests receive the same code as frontend tests.

  • conftest.py: Sets up useful fixtures for the pytest test runner.

  • coverage/: Output directory for test coverage commadns, npm run coverage_frontend and npm run coverage_backend. Will contain browsable HTML for both, after running these commands.

  • data/: Source files for the data stored in the backend. In particular, named SMARTS libraries.

  • docs/: Source files to build the documentation from, using the Sphinx documentation generator. Contains the file this developer guide was created from.

  • .flaskenv: Allows Flask to find the SMARTSexplore application when executing a command like flask run. Requires the python-dotenv package (which is listed as a dependency in setup.py) to work.

  • .gitignore: Typical gitignore file with a few manual adjustments.

  • instance/: The instance folder for the Flask application.

  • jest.config.js: Configure the Jest JavaScript test framework.

  • jest.setup.js: A setup file to run before the Jest tests.

  • package.json: Configuration file for npm. Contains all required frontend and JavaScript build dependencies as well as many useful build commands.

  • package-lock.json: See here.

  • profiling/: Code for profiling the (frontend) application by automating a Chrome browser + a Jupyter notebook for plotting that data.

  • pytest.ini: Configures the pytest test runner.

  • README.md: The readme file, containing simple setup instructions and a short overview.

  • setup.py: Configuration file for setuptools. Defines the SMARTSexplore Python package, Python dependencies, and useful commands for package building and distribution. Also exposes a command for building the Sphinx documentation.

  • smartsexplore/: Contains the application code, both for the backend and the frontend. Backend code is spread across Python module directories and files, the frontend code is contained in the smartsexplore/frontend/ subfolder.

  • tests/: Contains backend, frontend, and end-to-end test code.

  • webpack.common.js: The base configuration file for the Webpack build tool. All other webpack.*.js files use this file as a basis.

  • webpack.dev.js: Webpack configuration for development. Used by npm run watch.

  • webpack.e2e-test.js: Webpack configuration for executing end-to-end tests. Used by npm run _e2e_testprep and all npm commands based on that.

  • webpack.prod.js: Webpack configuration for production. Used by npm run build.

Code guidelines

  • Write all code using spaces for indentation.

  • For frontend JavaScript code:

    • Use ES6 modules.

    • Try to import external libraries as ES6 modules whenever possible.

    • Try to include external libraries by installing them as Node development dependencies, e.g. npm install d3-force --save-dev, whenever possible.

    • Import CSS files with import syntax too – Webpack will handle it for you.

  • For backend Python code, try to adhere to PEP8 whenever possible.