Solr

Compatibility notice

This documentation requires the following mStudio features which might not be available in all hosting plans: Container hosting.

What is Solr?

Apache Solr is a highly reliable, scalable and fault-tolerant search platform providing distributed indexing, replication, load-balanced querying, automated failover and recovery, centralized configuration, and more. Solr powers the search and navigation features of many of the world's largest internet sites, enabling powerful full-text search along with hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, and rich document handling.

Solr is built on Apache Lucene, a high-performance, full-featured text search engine library written in Java. Solr extends Lucene with HTTP APIs, making it easy to use from virtually any programming language. Solr's external configuration allows it to be tailored to many types of applications without Java coding, and it has a plugin architecture to support more advanced customization.

For content management systems like TYPO3, Solr provides powerful search capabilities that can significantly enhance the user experience by enabling fast and relevant search results across your website content.

Creating a Solr database

You can provision a Solr search engine in your mittwald hosting environment using containers. There are several approaches to set up Solr:

Using Terraform (Recommended)

The most convenient way to provision a Solr database is using Terraform with our Solr module. The following example shows how you can use this module in your own Terraform deployment:

module "solr" {
  source = "mittwald/solr/mittwald"

  project_id = var.project_id

  solr_version = "9"
  solr_core_name = "typo3"
  solr_heap = "2g"
}

output "solr_url" {
  value = module.solr.url
}

The solr_version input can be any valid Docker tag from the official solr repository. Use 9 to simply use the latest available version of Solr 9, or a more specific tag like 9.9.

After this, you can access your Solr backend using the URL returned by the Terraform output. Usually, the URL should be http://solr:8983. Note that this URL is only accessible from within the hosting environment (e.g., from your application containers or via SSH).

To access your Solr container from your local machine (without connecting it to a domain), you can use the mw container port-forward command:

$ mw container port-forward solr

While this command is running, you should be able to connect to your Solr instance via the URL http://localhost:8983.

TYPO3-Specific Terraform Setup

For TYPO3 projects, we provide a specialized Terraform module that automatically configures Solr with the necessary schema and configuration files optimized for TYPO3:

module "solr" {
  source = "mittwald/solr/mittwald"

  project_id = var.project_id

  solr_version = "9"
  solr_core_name = "typo3"
  solr_heap = "2g"
}

module "solr_typo3" {
  source = "mittwald/solr/mittwald//modules/typo3"

  solr_stack_id = module.solr.stack_id
  solr_core_name = "typo3"
  solr_core_language = "english"
  typo3_solr_version = "13.0.3"
}

output "solr_url" {
  value = module.solr.url
}

For multi-language TYPO3 installations, you can create separate cores for each language:

module "solr" {
  source = "mittwald/solr/mittwald"

  project_id = var.project_id
  solr_version = "9"
  solr_heap = "2g"
}

module "solr_typo3" {
  source = "mittwald/solr/mittwald//modules/typo3"
  for_each = toset(["german", "english"])

  solr_stack_id = module.solr.stack_id
  solr_core_name = "typo3_${each.key}"
  solr_core_language = each.key
  typo3_solr_version = "13.0.3"
}

output "solr_url" {
  value = module.solr.url
}

The TYPO3 module automatically:

• Downloads and installs the appropriate schema.xml for the specified language
• Configures language-specific analyzers and filters
• Sets up the required TYPO3 Solr plugin JAR file
• Creates core.properties with the correct configuration
• Installs additional configuration files like protwords, synonyms and stopwords

The typo3_solr_version should match the version of the TYPO3 Solr extension you're using in your project.

Using the mStudio UI

Alternatively, you can set up a Solr container manually:

1. Go to Containers in your project in mStudio and create a new one. You can choose any name you like.

2. As image, enter either solr:9 or a more specific version number like solr:9.9, depending on how specific the version should be constrained. You can keep the entrypoint and command as suggested.

3. (optional, but recommended) To automatically create a Solr core on container startup, change the command to solr-precreate <corename>, replacing <corename> with a name of your choice.

4. To make your Solr data persistent, create a volume under Volumes as follows:

   • Volume: Create new volume
   • Path in container (Mount Point): /var/solr

5. Set the following environment variables for the container:

   # adjust as needed
   SOLR_HEAP=512m
   SOLR_CONFIG_LIB_ENABLED=true
   SOLR_MODULES=scripting,analytics,analysis-extras,langid,clustering,extraction

The suggested port values are standard in the Solr context and can be accepted as is. The default port for Solr is 8983.

Using the CLI with mw container run

You can also deploy a Solr container using the mittwald CLI with the mw container run command:

mw container run \
  --name solr \
  --env SOLR_HEAP=512m \
  --env SOLR_CONFIG_LIB_ENABLED=true \
  --env SOLR_MODULES=scripting,analytics,analysis-extras,langid,clustering,extraction \
  --volume solr-data:/var/solr \
  --publish 8983:8983 \
  --create-volumes \
  solr:9.9 solr-precreate typo3

This command creates a new container named "solr" using the Solr image, sets the necessary environment variables including enabling the lib directory configuration and additional Solr modules, and mounts a volume for persistent data storage.

Using the CLI with mw stack deploy

If you prefer using Docker Compose, you can create a docker-compose.yml file and deploy it using the mw stack deploy command:

1. Create a docker-compose.yml file with the following content:

   version: "3"
   services:
     solr:
       image: solr:9.9
       command: solr-precreate typo3
       environment:
         - SOLR_HEAP=512m
         - SOLR_CONFIG_LIB_ENABLED=true
         - SOLR_MODULES=scripting,analytics,analysis-extras,langid,clustering,extraction
       ports:
         - "8983:8983"
       volumes:
         - solr-data:/var/solr
   volumes:
     solr-data:

2. Deploy the stack using the CLI:

   mw stack deploy

This approach is particularly useful when you need to deploy multiple containers that work together.

Accessing Solr

Accessing your container within your hosting environment

Once the container is running, you can verify the instance is accessible by requesting it via curl:

curl -X GET "http://solr:8983/solr/admin/info/system"

You should receive a response with information about your Solr instance, including version, memory usage, and system properties.

Accessing Solr from outside the hosting environment

The Solr URL (http://solr:8983) is only accessible from within the hosting environment. To access Solr from your local machine or from the internet, you have two options:

Option 1: Using port forwarding

You can use the mittwald CLI to forward the Solr port to your local machine:

mw container port-forward solr 8983:8983

This command creates a secure tunnel between your local machine and the Solr container. After running this command, you can access the Solr admin interface at http://localhost:8983/ in your local web browser.

Option 2: Connecting a domain

For more permanent access, you can connect a domain to your Solr container:

1. In mStudio, go to Domains and add a new domain or subdomain (e.g., solr.yourdomain.com)
2. Configure the domain to point to your Solr container on port 8983
3. Once the DNS has propagated, you can access Solr at http://solr.yourdomain.com/

Note that if you choose this option, you should consider implementing proper authentication to protect your Solr instance from unauthorized access.

Creating a Solr Core

If you did not create a core on container startup (using the solr-precreate command mentioned above), you'll need to create a core to store your search indices, after setting up your Solr container. You can do this using the Solr Admin UI or via the command line:

Using the Command Line

The easiest way to create a new Solr core is using the solr create_core command and the mw container exec command:

$ mw container exec <CONTAINER-ID> 'solr create_core -c typo3'

Initializing a TYPO3 Solr Core Manually

If you're not using the Terraform TYPO3 module, you can manually initialize a Solr core with TYPO3-specific configuration files using the mw container cp command. This approach is similar to using docker cp:

1. First, clone the TYPO3 Solr extension repository to get all configuration files:

   # Clone the TYPO3 Solr extension repository
   git clone https://github.com/TYPO3-Solr/ext-solr.git /tmp/ext-solr
   
   # Navigate to the configuration directory
   cd /tmp/ext-solr/Resources/Private/Solr/configsets/ext_solr_13_0_0/conf

2. Create the core directory structure in the Solr container:

   mw container exec <CONTAINER-ID> 'mkdir -p /var/solr/data/typo3/conf'
   mw container exec <CONTAINER-ID> 'mkdir -p /var/solr/data/typo3/lib'

3. Copy the configuration files to the Solr container:

   # Copy main configuration files
   mw container cp -r /tmp/ext-solr/Resources/Private/Solr/configsets/ext_solr_13_0_0/conf <CONTAINER-ID>:/var/solr/data/typo3/conf
   
   # Copy language-specific files (example for English)
   mw container cp -r /tmp/ext-solr/Resources/Private/Solr/configsets/ext_solr_13_0_0/conf/english <CONTAINER-ID>:/var/solr/data/typo3/conf/english
   
   # Copy the TYPO3 Solr plugin JAR file
   mw container cp /tmp/ext-solr/Resources/Private/Solr/typo3lib/solr-typo3-plugin-6.0.0.jar <CONTAINER-ID>:/var/solr/data/typo3/lib/

4. Create the core.properties file:

   mw container exec <CONTAINER-ID> 'echo "name=typo3" > /var/solr/data/typo3/core.properties'
   mw container exec <CONTAINER-ID> 'echo "config=solrconfig.xml" >> /var/solr/data/typo3/core.properties'
   mw container exec <CONTAINER-ID> 'echo "schema=english/schema.xml" >> /var/solr/data/typo3/core.properties'

5. Reload the Solr core to apply the new configuration:

   mw container exec <CONTAINER-ID> 'curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=typo3"'

6. Clean up the temporary files:

   rm -rf /tmp/ext-solr

For other languages, replace english in the file paths with the appropriate language directory (e.g., german, french, etc.). Replace <CONTAINER-ID> with your actual Solr container ID, which you can find using mw container list.

Using the Solr Admin UI

Alternatively, you can use the Solr admin UI to create a new core. For a TYPO3 Solr core, first follow steps 1-3 from the manual initialization section above to prepare all the necessary TYPO3 configuration files and the plugin JAR file.

After the prerequisites are met, you can create the new core via the Solr Admin UI:

1. Access the Solr Admin UI using one of the methods described in Accessing Solr from outside the hosting environment
2. Click on the "Core Admin" menu item
3. Click on "Add Core" and fill in the required fields:

• name: The name of your core (e.g., "typo3")
• instanceDir: /var/solr/data/typo3 (or your chosen core directory)
• dataDir: Leave as default or specify a custom directory
• config: solrconfig.xml
• schema: schema.xml

4. Click "Add Core" to create the TYPO3 Solr core

Operation

Solr runs as long as the container is running. When using Solr as a search engine for your CMS, the search indices need to be populated by your application. For this, many CMS platforms offer integrations.

The volume is automatically included in the regular backup of your project.

Setting up Solr with TYPO3 CMS

TYPO3 CMS can be integrated with Solr using the extension solr. This section provides a step-by-step guide to configure TYPO3 with your Solr instance.

Installing the Solr Extension

1. Install the Solr extension using Composer:

   composer require apache-solr-for-typo3/solr

   Or download it from the TYPO3 Extension Repository and install it through the Extension Manager.

2. After installation, go to the Extension Manager in the TYPO3 backend and activate the extension.

Configuring the Solr Connection

1. Create or edit your site configuration in TYPO3 (Site Management > Sites).

2. In the site configuration, go to the "Solr" tab and configure the connection:

   • Scheme and port: http and 8983
   • Host: solr (the hostname of your Solr container)
   • Port: 8983 (the default Solr port)
   • URL path to Apache solr server: /

3. Then, per language of your site configuration, configure the name of the Solr core. If your site has multiple languages, you should create one core per language.

   • Corename: typo3 (or the name of the core you created)

4. Save the configuration.

Configuring TypoScript

To get started quickly, include the static TypoScript templates that are provided by the extension. Refer to the extension's manual for more information.

Initializing the Index

For initializing the search index for the first time, refer to the extension manual.