Searching one chemical at a time (single-substance search)

In single-substance search, the user enters a full or partial chemical identifier (name, CASRN, InChiKey, or DSSTox ID) into a search box on the CCD homepage. Autocomplete can provide a list of possible matches. Figure 1 shows an example: the CCD landing page for the chemical Bisphenol A. This page is generated for Bisphenol A and links to but does not represent other chemicals. Each chemical in the CCD has its own page similar to this, with varying levels of accessible based on the information that is available.

Figure 1: Bisphenol A CCD landing page

The different domains of data available for this chemical are shown by the tabs on the left side of the page: for example, “Physchem Prop.” (physico-chemical properties), “Env. Fate/Transport” (environmental fate and transport data), and “Hazard Data” (in vivo hazard and toxicity data), among others.

Batch search

In batch search, the user enters a list of search inputs, separated by new lines, into a search box. The user selects the type(s) of inputs by selecting one or more checkboxes – include chemical identifiers, monoisotopic masses, or molecular formulas. Then, the user selects “Display All Chemicals” to display the list of substances matching the batch-search inputs, or “Choose Export Options” to choose options for exporting the batch-search results as a spreadsheet. The exported spreadsheet may include data from most of the domains available on an individual substance’s CCD page.

Figure 2: CCD Batch Search

The user can download the selected information in various formats, such as Excel (.xlsx), comma-separated values (.csv), or different types of chemical table files (.e.g, MOL).

Figure 3: CCD Batch Search Customized Export

The web interface for batch search only allows input of 10,000 identifiers at a time. If a user wants to retrieve information for more than 10,000 chemicals, identifiers will need to be separated into multiple batches and searched separately.

Challenges of web-based dashboard search

Practicing researchers may follow a workflow that looks something like this:

1. Start with a dataset that includes your chemical identifiers of interest. These may include chemical names, Chemical Abstract Service Registry Numbers (CASRNs), Distributed Searchable Structure-Toxicity Database (DSSTox) identifiers, or InChIKeys.
2. Export the chemical identifiers to a spreadsheet. Often, this is done by importing the data into an environment such as R or Python, in order to do some data wrangling (e.g., to select only the unique substance identfiers; to clean up improperly-formatted CASRNs; etc.). Then, the identifiers are saved in a spreadsheet (an Excel, .csv, or .txt file), one chemical identifier per row.
3. Copy and paste the chemical identifiers from the spreadsheet into the CCD Batch Search box. If there are more than 10,000 total chemical identifiers, divide them into batches of 10,000 or less, and search each batch separately.
4. Choose your desired export options on the CCD Batch Search page.
5. Download the exported spreadsheet of CCD data. By default, the downloaded spreadsheet will be given a file name that includes the timestamp of the download.
6. Repeat steps 3-5 for each batch of 10,000 identifiers produced in step 2.
7. Import the downloaded spreadsheet(s) of CCD data into the analysis tool you are using (e.g. R or Python).
8. Merge the table(s) of downloaded CCD data with your original dataset of interest.
9. Proceed with research-related data analysis using the chemical data downloaded from the CCD (e.g., statistical modeling, visualization, etc.)

Because each of these workflow steps requires manual interaction with the search and download process, the risk of human error inevitably creeps in. Here are a few real-world possibilities (the authors can neither confirm nor deny that they have personally committed any of these errors):

• Researchers could copy/paste the wrong identifiers into the CCD batch search, especially if more than 10,000 identifiers need to be divided into separate batches.
• Chemical identifiers could be corrupted during the process of exporting to a spreadsheet.
  • If a researcher opens and resaves a CSV file using Microsoft Excel, any information that appears to be in date-like format will be automatically converted to a date (unless the researcher has the most recently-updated version of Excel and has selected the option in Settings that will stop Excel from auto-detecting dates). This behavior has long been identified as a problem in genomics, where gene names can appear date-like to Excel (Abeysooriya et al. 2021). It also affects cheminformatics, where chemical identifiers can appear date-like to Excel. For example, the valid CASRN “1990-07-4” would automatically be converted to “07/04/1990” (if Excel is set to use MM/DD/YYYY date formats). CCD batch search cannot recognize “07/04/1990” as a valid chemical identifier and will be unable to return any chemical data.
• Researchers could accidentally rename a downloaded CCD data file to overwrite a previous download, e.g. when searching multiple batches of identifiers.
• Researchers could mistakenly import the wrong CCD download file back into their analysis environment e.g. when searching multiple batches of identifiers.

Moreover, the manual stages of this kind of workflow are also non-transparent and not easily reproducible. Utilizing APIs for data exploration and retrieval can alleviate these concerns.

Authentication

Authentication, found in upper left tab on each web interface page, is required to use the APIs. To authenticate themselves in the API web interface, the user must input their unique API key.

Figure 5: API Key Authentication

Request Construction

APIs effectively automate the process of accessing and downloading the data that populates the CCD. APIs do this via requests using the Hypertext Transfer Protocol (HTTP) that enables communication between clients (e.g. your computer) and servers (e.g. the CCD).

In the CTX API web interface, the colored boxes next to each endpoint indicate the type of the associated HTTP method. GET is used to request data from a specific web resource (e.g. a specific URL); POST is used to send data to a server to create or update a web resource. For the CTX APIs, POST requests are used to perform multiple (batch) searches in a single API call; GET requests are used for non-batch searches.

You do not need to understand the details of POST and GET requests in order to use the API. Let’s consider constructing an API request to Get data by dtxsid under the Chemical Details Resource.

Figure 6: Get Details by DTXSID

The web interface has two subheadings:

• Path Parameters contain user-specified parameters that are required in order to tell the API what URL (web address) to access. In this case, the required parameter is a string for the DTXSID identifying the chemical to be searched.
• Query-String Parameters contain user-specific parameters (usually optional) that tell the API what specific type(s) of information to download from the specified URL. In this case, the optional parameter is a projection parameter, a string that can take one of five values (chemicaldetailall, chemicaldetailstandard, chemicalidentifier, chemicalstructure, ntatoolkit). Depending on the value of this string, the API can return different sets of information about the chemical. If the projection parameter is left blank, then a default set of chemical information is returned.

The default return format is displayed below and includes a variety of fields with data types represented.

Figure 7: Get Details by DTXSID, Return Format

Pictured below is an example of returned Details for Bisphenol A with the chemicaldetailstandard value for projection selected.

Figure 8: Returned Details for Bisphenol A

Package Settings

Users can run library(ctxR) to install from CRAN or install the development version of ctxR like so:

if (!library(devtools, logical.return = TRUE)){
  install.packages(devtools)
  library(devtools)}

devtools::install_github("USEPA/ctxR")

API Key Storage

As previously described, a user must have an API key to use in order to access the CTX APIs. A FREE API key can be obtained by emailing the CTX API Admins. In the example code, the API key will be stored as the variable my_key.

my_key <- 'YOUR_CTX_API_key'

For general use of the package, the user may use the function register_ctx_api_key() to store the API key in the current session or more permanently for access across sessions.

# This stores the key in the current session
register_ctx_api_key(key = '<YOUR API KEY>')

# This stores the key across multiple sessions and only needs to be run once. If the key changes, rerun this with the new key.
register_ctx_api_key(key = '<YOUR API KEY>', write = TRUE)

Once the API key is stored, the default display setting is turned off for protection. To change this, use the following functions as demonstrated.

# To show the API key
ctxR_show_api_key()
getOption('ctxR')$display_api_key

# To hide the API key
ctxR_hide_api_key()
getOption('ctxR')$display_api_key

Finally, to access the key, use the ctx_key() function.

ctx_key()

Chemical API

In this section, several ctxR functions are used to access different types of information from the CTX Chemical API.

bpa_details <- get_chemical_details(DTXSID = 'DTXSID7020182')

bpa_info <- get_chem_info(DTXSID = "DTXSID7020182")

bpa_info_experimental <- get_chem_info(DTXSID = "DTXSID7020182", type = 'experimental')

Hazard API

In this section, several ctxR functions are used to access different types of information from the CTX Hazard API.

bpa_hazard <- get_hazard_by_dtxsid(DTXSID = 'DTXSID7020182')

bpa_human_hazard <- get_human_hazard_by_dtxsid(DTXSID = 'DTXSID7020182')

bpa_eco_hazard <- get_ecotox_hazard_by_dtxsid(DTXSID = 'DTXSID7020182')

Bioactivity API

In this section, several ctxR functions are used to access different types of information from the CTX Bioactivity API.

bpa_bioactivity <- get_bioactivity_details(DTXSID = 'DTXSID7020182')

assay_id_search <- get_bioactivity_details(AEID = 42)