SOLR Reports
This article describes how to generally set up SOLR reports and any problems that can arise in this context can be solved.
Introduction
On customer systems with large data sets complex searches like the catalogue search or reports like course progress reports need lots of system resources / take longer time to load. To improve performance, it was necessary to store the data redundantly in a structure which is more suitable for the specific search.
We decided to use the SOLR technology because it uses concepts of internet searches which results in even faster response times compared to storing the data redundatly in a more suitable schema just in the database.
SOLR Indices
As of this writing we have, 9 SOLR indices, three for normal searches and six for reports. Since the data in the database is constantly changing the SOLR indices need to be actualized regularly.
Search-indices: catalogindex, personindex, mediaindex
Report-indices: courseattendanceindex, courselearninghistoryindex, courseprogressindex, coursescostatusindex, feedbackquestionsindex, testquestionsindex
The search indices are relativ small and a complete rebuild takes only a few minutes. The report indices are often pretty big and a full build can take multiple hours. The actualization of the search indices is therefore always done with the technically easy way of a full new build. The actualization of the report indexes is then done by applying just these changes on the SOLR report index which also only takes a few minutes.
SOLR Manager
The two search indices catalogindex and personindex are active by default for all customers. The other indices (mediaindex and the report indexes) are not in use by most customers. To avoid unnecessary load on the database server only the SOLR indices which are used should be built and actualized.
Therefore we have the SOLR navigation point SOLR Indices to activate and deactivate SOLR indexes. By default, only catalogindex and personindex are active. Only active indices are built or actualized when the relevant SOLR job runs. By default the SOLR Manager (navigation point SOLR Indices) is not available and first needs to be configured.
SOLR Jobs
Search-Indexes: CreateSolrIndex, CheckSolrAvailability
CreateSolrIndex: Performs a complete rebuilt of the activated search indexes. In most cases it is configured to run every half hour or hour.
CheckSolrAvailability: This job which is scheduled to run at five minute boundaries was created to ensure a quick rebuild of possibly lost search indices during a deployment.
Report-Indizes: SolrIndexFullBuild, SolrIndexActualization
SolrIndexFullBuild: Performs a complete rebuilt for all active report indices which are currently not available meaning which have never been built before or for which a new full build was initiated via Solr navigation as described above. There are some technical differences to search indices and problems with the Quartz Scheduler which require a separate job. It is especially important to configure the schedule interval large enough to be sure the job is completed before it runs again. So it should be configured to run just once a day.
SolrIndexActualization: It actualizes active report indices which are currently available meaning for which a full build completed already successfully and there was no initiation of a new full build for this index in the meantime. The job applies the changes which took place in the database since the last run of the actualization job or the begin of the last full build if first run after a full build. It should be configured to run automatically every 15 minutes to keep index data pretty current and keep actualization run times short.
Initial setup for report indices and configuration of SOLR reports and cronjobs needs to be performed by imc. If you have questions or are interested in using SOLR reports, or you experience longer load times on any of the standard reports mentioned above, raise a request ticket in our Service Desk and we will get in touch.
Troubleshooting for Customers Using SOLR Reporting
Error: The SOLR index is currently not available. Please try again later.
Troubleshooting steps:
For SOLR indices, the best course of action to troubleshoot is to go into Support Information and download (2) the log files (1).
You will then need to see if there are any errors with the scheduled jobs.
In the log files search for de.imc.clix.core.solr.SolrIndexFullBuildJob.
You might notice a couple of things:
1. the job is either still running OR
2. the job shows an error message after startup
Either way, please raise an incident ticket in our Service Desk, provide all information and logs so we can troubleshoot further with our DEV and hosting team if needed.
Error: Data is not up-to-date - i.e. the data (e.g. enrolment status or personal end date) in the SOLR report does not match the user data in the participant administration of a course.
Troubleshooting steps:
- Download the log files from the Support Information.
- Check if the job (de.imc.clix.core.solr.SolrIndexActualizationJob) has run successfully or if it has produced errors.
Either way, please raise an incident ticket in our Service Desk, provide all information and logs so we can troubleshoot further with our DEV and hosting team if needed.