.. _db:

Database
========

CVE-Search makes use of multiple databases, these are:

* MongoDB
* Redis

MongoDB
#######

The MongoDB database is (by default) called cvedb and has 11 collections:

* cves (Common Vulnerabilities and Exposure items) - source NVD NIST (API)
* cpe (Common Platform Enumeration items) - source NVD NIST
* cwe (Common Weakness Enumeration items) - source NVD NIST
* capec (Common Attack Pattern Enumeration and Classification) - source NVD NIST
* ranking (ranking rules per group) - local cve-search
* `MITRE Reference Key/Maps <https://cve.mitre.org/data/refs/>`_ - source MITRE reference Key/Maps
* info (metadata of each collection like last-modified) - local cve-search
* via4 `VIA4CVE <https://github.com/cve-search/VIA4CVE>`_ cross-references, and has 3 additional sources:
    * `MITRE Reference Key/Maps <https://cve.mitre.org/data/refs/>`_.
    * Red Hat RPM to CVE database.
    * Red Hat RHSA Oval database.

The initial setup of CVE-Search happens only once, at the installation.
This consists of two steps and one optional step.

 * Populating the database
 * *Optional:* You can also run the "Other CPE Dictionary" script to help fill in the blanks
 * Updating the database

.. _pop_db:

Connecting to MongoDB
---------------------

MongoDB has two possible syntax for connecting to the database.

* `mongodb://` - Default
* `mongodb+srv://`

The default syntax allows for connectivity to a single host or a replica set.  The SRV syntax
allows for connecting using a  single DNS hostname which seeds multiple hosts in a replica set.
The SRV DNS record contains all of the details required for connecting to any server contained
in a replia set, even if one of the nodes is unavailable.

To enable the SRV scheme, set the variable `DnsSrvRecord` to `True` in the configuration.ini file.
For more information, read `MongoDB 3.6: Here to SRV you with easier replica set connections <https://www.mongodb.com/developer/article/srv-connection-strings/>`_.

*Note:* MongoDB Atlas requires the use of the SRV syntax.

Database User Authentication
----------------------------

When passing a username and password, CVE-Search submits the values against the default `admin` 
database. If the authentication information is stored in a database other than `admin`, 
authentication attempts will fail.

To change the default authentiation database, set the variable `AuthDB` in the configuration.ini file.

Populating the database
-----------------------

For the initial run, you need to populate the CVE database by running:

.. code-block:: bash

    ./sbin/db_mgmt_cpe_dictionary.py -p
    ./sbin/db_mgmt_json.py -p
    ./sbin/db_updater.py -c

It will fetch all the existing Common Vulnerabilities and Exposures (CVE) & Common Platform Enumeration (CPE) data from NVD NIST API.
The initial import (during `db_mgmt_json.py -p`) might take some time depending on your configuration, e.g., over 45 minutes. Please be patient.

These could be also run as a SystemD service. Example units are under `_etc/systemd/system/`:
`cvesearch.db_init.service <https://github.com/cve-search/cve-search/blob/master/_etc/systemd/system/cvesearch.db_init.service>`_ &
`cvesearch.db_init.target <https://github.com/cve-search/cve-search/blob/master/_etc/systemd/system/cvesearch.db_init.target>`_.

.. code-block:: bash

    sudo systemctl start --no-block cvesearch.db_init.target


If you want to add the cross-references from NIST, Red Hat and other vendors thanks to `VIA4CVE <https://github.com/cve-search/VIA4CVE>`_:

.. code-block:: bash

    ./sbin/db_mgmt_ref.py

NB: If you want to  import your own JSON from VIA4CVE, you have to replace URL in sources.ini the VIA4 attribute with
`file:///PATH/TO/VIA4CVE/VIA4CVE-feed.json`.

.. _upd_db:

Updating the database
---------------------
An updater script helps to start the db_mgmt_*

.. code-block:: bash

    ./sbin/db_updater.py

You can run it in a crontab, logging is done in log/update_populate.log by default.

These could be also run as a SystemD service and a timer that automates regular updates. Example units are under `_etc/systemd/system/`:
`cvesearch.db_updater.service <https://github.com/cve-search/cve-search/blob/master/_etc/systemd/system/cvesearch.db_updater.service>`_ &
`cvesearch.db_updater.timer <https://github.com/cve-search/cve-search/blob/master/_etc/systemd/system/cvesearch.db_updater.timer>`_.

.. code-block:: bash

    sudo systemctl start cvesearch.db_updater.timer
    sudo systemctl enable cvesearch.db_updater.timer


.. _repop_db:

Repopulating the database
-------------------------
To easily drop and re-populate all the databases

.. code-block:: bash

    ./sbin/db_updater.py -f

This will drop all the existing external sources and reimport everything. This operation can take some time
and it's usually only required when new attributes parsing are added in cve-search.

This could be also run as a SystemD service. Example units are under `_etc/systemd/system/`:
`cvesearch.db_repopulate.service <https://github.com/cve-search/cve-search/blob/master/_etc/systemd/system/cvesearch.db_repopulate.service>`_ &
`cvesearch.db_repopulate.target <https://github.com/cve-search/cve-search/blob/master/_etc/systemd/system/cvesearch.db_repopulate.target>`_.
Using the service will stop `cvesearch.web.service <https://github.com/cve-search/cve-search/blob/master/_etc/systemd/system/cvesearch.web.service>`_
during the repopulation. This becomes handy as the web GUI & CVE-Search API would give errors or incomplete data during the process.

.. code-block:: bash

    sudo systemctl start --no-block cvesearch.db_repopulate.target


Redis
#####

3 Redis databases are used:

* Database number 10: The cpe (Common Platform Enumeration) cache - source MongoDB cvedb collection cpe
* Database number 11: The notification database - source cve-search
* Database number 12: The `CVE reference database <https://cve.mitre.org/data/refs/>`_ is a cross-reference database to CVE IDs against various vendors ID - source NVD NIST/MITRE


Populating the database
-----------------------
Depending on the switches that are appended to the `./sbin/db_updater.py` command, the redis database will be populated.
Check the help of that specific script for further details.