Upgrading from v9 to v10¶
Prerequisites¶
The steps listed in this article require an existing local installation of InvenioRDM v9, please make sure that this is given!
If unsure, run invenio-cli install
from inside the instance directory before executing the listed steps.
Backup
Always backup your database and files before you try to perform an upgrade.
Older versions
In case you have an InvenioRDM installation older than v9, you can gradually upgrade using the existing instructions to v9 and afterwards continue from here.
Upgrade Steps¶
Upgrade your invenio-cli
Make sure you have the latest invenio-cli
, for InvenioRDM v10 the release is v1.0.8.
$ invenio-cli --version
invenio-cli, version 1.0.8
For local installations, make sure that you prefix each invenio
command with pipenv run
.
Elasticsearch check¶
InvenioRDM v10.0 removes support for Elasticsearch v6. If you are using it, you will need first to migrate to Elasticsearch v7 or directly to OpenSearch, before upgrading.
Elasticsearch v7 change of license
Do not upgrade to Elasticsearch v7.11 or newer, due to the change of license.
The last version with Apache 2.0
license is 7.10.2.
Installing the Latest Versions¶
If you are planning to migrate from Elasticsearch v7 to OpenSearch, follow the step below. Otherwise, skip to the next section Upgrade InvenioRDM.
We strongly recommend migrating or planning to migrate to OpenSearch. Elasticsearch support will be removed in a future version of InvenioRDM.
Migrate to OpenSearch¶
Elasticsearch v7 and OpenSearch v1 are very similar and the migration is rather simple (see the official OpenSearch documentation here). For OpenSearch v2, the only breaking change affecting InvenioRDM, is the removal of doc_type
(you can read more here). With this release, you can safely migrate to OpenSearch v2, skipping OpenSearch v1.
OpenSearch v1 Log4Shell security vulnerability
If you have decided to use OpenSearch v1, make sure you install a version greater or equal to v1.2.1
: previous versions are affected by the Log4Shell security vulnerability.
-
Prepare your new OpenSearch v2/v1 cluster.
You can read more on how to setup an OpenSearch cluster in the official documentation. Invenio Helm Charts have been updated adding support for OpenSearch, but the configuration will setup a demo cluster and it should not be used in production. OpenSearch provides official Helm charts suitable for production environments.
For local development, make sure that you have migrated to OpenSearch in your Docker files in your instance:
- In
docker-services.yml
, replace thees
andkibana
sections withopensearch
andopensearch-dashboards
sections, as in here. - In
docker-compose.yml
, replace thees
andkibana
keywords withopensearch
andopensearch-dashboards
keywords, as in here. - In
docker-compose.full.yml
, replace thees
andkibana
keywords withopensearch
andopensearch-dashboards
keywords, as in here.
- In
-
In your instance, edit the
Pipfile
and replaceinvenio-app-rdm = {extras = [... "elasticsearch7"]...
withopensearch2
(oropensearch1
). For example, in thePipfile
:[packages] -invenio-app-rdm = {extras = ["postgresql", "elasticsearch7"], version = "~=9.1.0"} +invenio-app-rdm = {extras = ["postgresql", "opensearch2"], version = "~=9.1.0"}
-
In your instance, edit the
.invenio
file:-elasticsearch = 7 +search = opensearch2
-
Do you have any customisation or custom module that contains Elasticsearch v7 mappings?
If not, go to the next step, step 5.
If yes, make sure that you copy the
v7
folder and paste it in the same location, with the new nameos-v2
(oros-v1
). The final folder structure should look like this (do not forget the__init__.py
files, they are required!):my-custom-module |-- mappings |-- __init__.py + |-- os-v2 + |-- __init__.py + |-- <record> + |-- <record>-v1.0.0.json |-- v7 |-- __init__.py |-- <record> |-- <record>-v1.0.0.json
OpenSearch v2 breaking change
Make sure that you have removed the
_type
field everywhere. OpenSearch v2 will fail if still present! -
In your
invenio.cfg
, if you have declared your Elasticsearch configuration:- Rename the
INVENIO_SEARCH_ELASTIC_HOSTS
configuration variable toINVENIO_SEARCH_HOSTS
. - Change its value to connect to the new cluster (for local development, the default is
localhost:9200
).
- Rename the
-
Uninstall the Elasticsearch Python libraries:
cd <my-site> pip uninstall -y elasticsearch elasticsearch-dsl
Upgrade InvenioRDM¶
Upgrade the RDM version:
cd <my-site>
# Upgrade to InvenioRDM v10
invenio-cli packages update 10.0.0
Optionally, update the file <my-site>/Pipfile
. This step is not necessary, but suggested for local development.
[packages]
---invenio-app-rdm = {extras = [...], version = "~=9.0.0"}
+++invenio-app-rdm = {extras = [...], version = "~=10.0.0"}
From now on, make sure that the services (database, search, etc.) are up and running. For a local development, run invenio-cli services start
.
Re-index data¶
Re-index data in the cluster:
# recreate indices
invenio index destroy --yes-i-know
invenio index init
# reindex records
invenio rdm-records rebuild-index
invenio communities rebuild-index
Re-build assets¶
Create the required folder for custom fields (even if empty) and rebuild the assets:
# Create custom fields folder
mkdir assets/templates/custom_fields
# Re-build the instance assets
invenio-cli assets build
Data Migration¶
Finally, you can run the database schema upgrade, and perform the records data migration:
# Perform the database migration
invenio alembic upgrade
# Run data migration script
invenio shell $(find $(pipenv --venv)/lib/*/site-packages/invenio_app_rdm -name migrate_9_0_to_10_0.py)
Complete re-indexing¶
Complete the re-indexing of the data in the search cluster. In a python shell (invenio-cli pyshell
), run:
from invenio_access.permissions import system_identity
from invenio_communities.proxies import current_communities
from invenio_records_resources.proxies import current_service_registry
from invenio_requests.proxies import current_events_service, current_requests_service
# reindex users
users_service = current_service_registry.get("users")
users_service.rebuild_index(system_identity)
# reindex groups
groups_service = current_service_registry.get("groups")
groups_service.rebuild_index(system_identity)
# reindex members and archived invitations
members_service = current_communities.service.members
members_service.rebuild_index(system_identity)
# reindex requests
for req_meta in current_requests_service.record_cls.model_cls.query.all():
request = current_requests_service.record_cls(req_meta.data, model=req_meta)
if not request.is_deleted:
current_requests_service.indexer.index(request)
# reindex requests events
for event_meta in current_events_service.record_cls.model_cls.query.all():
event = current_events_service.record_cls(event_meta.data, model=event_meta)
current_events_service.indexer.index(event)
Info
These commands are needed to make sure all record types are re-indexed. However, in future releases, a new simple command will include all of them.
Accessing the Administration panel¶
The new permission action administration-access
is needed to access the administration panel. There are two ways to allow users to access the new administration panel.
Allow specific users¶
# Add access to administration to an user by its email
invenio access allow administration-access user <user_email>
Allow by role¶
# Create a role
invenio roles create administrator
# Allow access to administration to the administrator role
invenio access allow administration-access role administrator
# Add administrator role to an user, by its emails
invenio roles add <user_email> administrator