Adding solr cloud service to babel by liseli · Pull Request #239 · hathitrust/babel

liseli · 2026-07-09T20:21:06Z

This PR is to integrate Solr 8 in the babel repository, with these changes we are ready to apply changes on ls and test them by accessing the Solr 8 cluster.

You will start up the services following the instructions in the README.md file.

Once Solr is up and running, access http://localhost:8981/solr/#/~cloud?view=graph, and you will see

The Solr cluster will start and create the collection, but we must manually index the sample data to run queries on the front end. I've created the ticket ETT-1678 to generate an image with the sample data, and then we can add these services to the Docker environment to automatically start Solr with data for testing.

Meanwhile, the process for indexing data is explained here. I've tested it, and it works, but you will need to clone the lss_solr_configs repository to your machine and follow the steps to run the script indexing_data.sh.

After indexing the sample data, you will run queries and see the results

eumalin · 2026-07-10T13:53:28Z

+    environment:
+      - ZK_HOST=zoo1:2181
+      - SOLR_OPTS=-XX:-UseLargePages
+    depends_on:


Not sure if depends_on: zoo1 is actually doing anything once it goes through extends: in the main compose file (Compose doesn't carry depends_on through extends, it has to be redeclared where you extend it). Does zoo1 come up before solr?

moseshll · 2026-07-14T16:18:14Z

In Solr Admin/ZK Status I get the warning

Could not execute ruok towards ZK host zoo1:2181. Add this line to the 'zoo.cfg' configuration file on each zookeeper node: '4lw.commands.whitelist=mntr,conf,ruok'. See also chapter 'Setting Up an External ZooKeeper Ensemble' in the Solr Reference Guide.

Not sure if this is expected.

Also, after performing the steps outlined in lss_solr_configs, core-x_shard1_replica_n1 shows 0 documents. Maybe its the ZK status thing. I guess it doesn't matter since one of the next steps is to have a sample index integrated so we don't have to do this manually.

moseshll

I was able to (re)index the sample and bring up Solr as described. Errors encountered in previous iteration are gone now. APPROVE

eumalin

LGTM

liseli · 2026-08-11T23:40:41Z

@aelkiss @moseshll @kron-spar and @eumalin, please check the changes I've applied to have access to Solr 6 and Solr 9. I've tested, and it is working fine with Solr 6; I'm not sure it works with Solr 9. Everything starts out fine with Solr 9, with no errors, but the queries return empty results. It could be related to an incompatibility with Perl code, not to these changes.

I have updated some config files in Babel to have access to Solr 6 and Solr 9. By default, when running the commands, the application will start with Solr 6.

docker compose --profile frontend build
docker compose --profile frontend up

Running it will start up and access Solr 9.

LSS_SOLR_HOST=solr-lss-dev-solr9 docker compose --profile backend --profile solr9 build
LSS_SOLR_HOST=solr-lss-dev-solr9 docker compose --profile backend --profile solr9 up

I have defined Solr6 and Solr 9 services in thecommon/docker/solr.yml and see below the list of the files I've updated too.

In ls/lib/Config/global.conf, the Solr 6 production source of truth (12-shard mbooks_solr_engines) is hardcoded. - This stays untouched
ls/lib/Config/local.conf seems not to be used, or I can not understand very well how it is used.
In ls/lib/Config/docker-local.conf, I have defined the environment variables LSS_SOLR_ENGINE_URL_A and LSS_SOLR_ENGINE_URL_B pointed to solr core-x and core-y, instead of hardcoding it, and is the way we can pass the URL specific for each version of Solr - LSS_SOLR_HOST is by default solr-lss-dev or defined as a parameter for solr 9 LSS_SOLR_HOST=solr-lss-dev-solr9. I decided not to create core-y for Solr 9, so we should check if, in the future, there are some related issues.
In the docker-compose.yml, the apache service uses common/docker/apache.yml, which defines the different environment variables used by the application, so I've added LSS_SOLR_ENGINE_URL_A and LSS_SOLR_ENGINE_URL_B there.
In the docker-compose.yml, perl-dev and perl-test use common/docker/perl.yml, so I have added LSS_SOLR_ENGINE_URL_A and LSS_SOLR_ENGINE_URL_B.
- It seems the Perl code only needs the Solr URL provided by the config file 000-default.conf
$config->get('mbooks_solr_engines') in slip-lib/Search/Searcher.pm.

The latest change was to add the variables to apache/000-default.conf, because docker-compose.yml/apache.yml sets LSS_SOLR_ENGINE_URL_A and LSS_SOLR_ENGINE_URL_B as container-level environment variables on the apache service, but Apache doesn't forward its own process environment to CGI scripts by default — only variables explicitly listed via SetEnv/PassEnv in the vhost config. I've seen that apache/000-default.conf has SetEnv lines for SDRROOT, SDRDATAROOT, PTSEARCH_SOLR, etc., so that was the main reason why I added LSS_SOLR_ENGINE_URL_A/_B, so the CGI process can see them.

I've seen some warnings in the Apache logs that I fixed too; please check if the fix makes sense

Error:

/htapps/babel/catalog/.htaccess: Invalid command 'php_value', perhaps misspelled or defined by a module not included in the server configuration

Fix:
Commenting out the line php_value magic_quotes_gpc false in catalog/.htaccess solved the issue. There seems to be some legacy configuration that is safe to remove; magic_quotes_gpc was removed in PHP 5.4.

Error:

[cgi:error] [pid 143:tid 143] [client 192.168.65.1:52378] AH01215: stderr from /htapps/babel/ping/cgi/ping: Use of uninitialized value $idp_url in substitution (s///) at /htapps/babel/mdp-lib/Institutions.pm line 333., referer: http://localhost:8080/Search/Home?adv=1&lookfor%5B%5D=Commissioner+of+Railroads+and+Telegraphs&type%5B%5D=title

Fix
It seems that for some MySQL queries, the template is NULL, so right now it defaults to an empty string ``

eumalin · 2026-08-12T16:24:41Z

@liseli Using Claude, I tried this out end to end. I brought up zoo1 and Solr 9, ran perl-dev and apache, and made a request through Apache to check the environment variables. Everything works: PassEnv is passing the Solr URLs through correctly, and local.conf is being used (docker-compose.yml copies docker-local.conf over it, so your edits do apply).

I think the empty results have two separate causes:

On a brand new Solr 9 setup, core-x has no documents in it yet (at least in my test).
core-y is never created, only core-x is. That matters more than it might seem: the code that reads the A/B URLs is not Searcher.pm, it's Facets.pm, it picks A or B at random for each search to spread out the load. So on Solr 9, about half of all searches would randomly try to hit core-y, which doesn't exist. That would show up as results being empty sometimes and not other times, even after you index data.

The Institutions.pm fix looks good to me, simple, safe way to stop the warning.

One thing I couldn't find: the catalog/.htaccess fix (commenting out php_value magic_quotes_gpc) isn't in this PR. Did that get committed somewhere else, or did it not make it in?

moseshll · 2026-08-12T16:41:18Z

I note that with a temp hack to the way ls/slip-lib handle basic auth, I can run a query but Solr responds with "ctrlnum is not a valid query field".

eumalin · 2026-08-12T17:06:04Z

@moseshll Good finding, that tracks. add_basic_auth in slip-lib/Search/Searcher.pm only sends an Authorization header if config key solr_basic_auth_token is set. This PR does not set it for the LSS/ls Solr connection. That means nothing gets sent and Solr 9 (auth enabled) rejects every query.

Production has the pattern for this on a different core (slip-lib/Config/run-20.conf:101): solr_basic_auth_token = ENV[PTSEARCH_SOLR_BASIC_AUTH]. The same fix should work here: add solr_basic_auth_token = ENV[LSS_SOLR_BASIC_AUTH] to docker-local.conf, set LSS_SOLR_BASIC_AUTH (base64 of admin:solrRocks) in apache.yml/perl.yml.

The ctrlnum error after that is probably a separate schema issue on the Solr 9 side. @liseli, it's worth checking whether core-x's config set matches the production mbooks schema.

moseshll · 2026-08-12T17:27:54Z

@eumalin I had identified an issue with ls/lib/LS/Searcher/Facets.pm over in ETT-1700, it should send the context $C to get the request object in __Solr_result, e.g., my $req = $self->__get_request_object($url, $C); so the mdp config is available to the LWP bits & pieces. That patch should probably be added to this PR.

I found that after I excised the fields ctrlnum, fullgeographic, fullgenre, and hlb3 from the dismax*.yaml files under ls/lib/Config I was able to get an acceptable query response. Right now I have the auth stuff hacked in, slip-lib/Search/Searcher.pm, e.g. $req->authorization_basic('solr', 'solrRocks'); just to get things moving.

Obviously the dismax yaml files need to be left intact for the time being, so I anticipate having e.g. dismax_A_Solr9.yaml with removed or renamed fields... or more likely a separate config dir or subdir for solr9. Or a separate AB_test_config. Not sure yet.

moseshll · 2026-08-12T18:32:02Z


        my $host = $ENV{'HTTP_HOST'} || 'localhost';
-        my $idp_url = $hash_ref->{template};
+        my $idp_url = $hash_ref->{template} // '';


This will help declutter the logs, because I am certain this is one of the sites of "uninitialized value" warnings cited in ETT-1776. This is the proper mitigation.

moseshll · 2026-08-12T19:35:52Z

In light of Lianet's recommended approach I am adding Solr6 and Solr9 directories under ls/lib/Config for the dismax_A and dismax_B files. I'm experimenting with a LSS_SOLR_ENGINE_VERSION ENV that will point to one of those. The main wrangling seems to be in the facet config, a few small changes should do it.

liseli · 2026-08-12T20:51:25Z

@eumalin and @moseshll, thank you so much for your comments and for reviewing this PR. After adding add_basic_auth to set up the authorization header, I can see query results from Solr 9. Right now, this PR seems to be in a good state to merge.

moseshll · 2026-08-13T16:33:40Z


    my $url = $self->__get_Solr_select_url($C, $query_string, $AB);
-    my $req = $self->__get_request_object($url);
+    my $req = $self->__get_request_object($url, $C);


So the mdp-config with Solr auth token can be retrieved.

aelkiss · 2026-08-13T18:04:26Z

    - SDRDATAROOT=/sdr1
+    - LSS_SOLR_ENGINE_URL_A=http://${LSS_SOLR_HOST:-solr-lss-dev}:8983/solr/core-x
+    - LSS_SOLR_ENGINE_URL_B=http://${LSS_SOLR_HOST:-solr-lss-dev}:8983/solr/core-y
+    - LSS_SOLR_BASIC_AUTH=YWRtaW46c29sclJvY2tz


May be worth a comment that this is the default 'admin:solrRocks' credential

aelkiss · 2026-08-13T18:06:44Z


        my $host = $ENV{'HTTP_HOST'} || 'localhost';
-        my $idp_url = $hash_ref->{template};
+        my $idp_url = $hash_ref->{template} // '';


@moseshll I thought we had determined a long time ago that template wasn't actually being used, and so we haven't been keeping it up to date or even populated at all for some institutions. Is this code running but the result is never used anywhere?

aelkiss · 2026-08-13T18:52:38Z

I'm able to run solr 9 and index documents into it, but if I try to query I get no results and this error in the error log:

solr-lss-dev-solr9  | 2026-08-13 18:43:12.146 INFO  (qtp549293029-19-null-66) [ t:null-66] o.a.s.s.SolrDispatchFilter Error authenticating =>java.lang.IllegalArgumentException: Illegal base64 character 5b
solr-lss-dev-solr9  | 	at java.base/java.util.Base64$Decoder.decode0(Unknown Source)
solr-lss-dev-solr9  | 
solr-lss-dev-solr9  | 2026-08-13 18:43:12.147 WARN  (qtp549293029-19) [] o.e.j.s.HttpChannel /solr/core-x/select =>org.apache.solr.common.SolrException: Error during request authentication, 
solr-lss-dev-solr9  | 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:354)

I'll poke around and see what I can figure out.

liseli · 2026-08-13T18:59:08Z

I'm able to run solr 9 and index documents into it, but if I try to query I get no results and this error in the error log:

solr-lss-dev-solr9  | 2026-08-13 18:43:12.146 INFO  (qtp549293029-19-null-66) [ t:null-66] o.a.s.s.SolrDispatchFilter Error authenticating =>java.lang.IllegalArgumentException: Illegal base64 character 5b
solr-lss-dev-solr9  | 	at java.base/java.util.Base64$Decoder.decode0(Unknown Source)
solr-lss-dev-solr9  | 
solr-lss-dev-solr9  | 2026-08-13 18:43:12.147 WARN  (qtp549293029-19) [] o.e.j.s.HttpChannel /solr/core-x/select =>org.apache.solr.common.SolrException: Error during request authentication, 
solr-lss-dev-solr9  | 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:354)

I'll poke around and see what I can figure out.

@aelkiss, I fixed this issue. Could you please pull the branch again? It should disappear.

liseli · 2026-08-13T19:00:59Z

I'm able to run solr 9 and index documents into it, but if I try to query I get no results and this error in the error log:

solr-lss-dev-solr9  | 2026-08-13 18:43:12.146 INFO  (qtp549293029-19-null-66) [ t:null-66] o.a.s.s.SolrDispatchFilter Error authenticating =>java.lang.IllegalArgumentException: Illegal base64 character 5b
solr-lss-dev-solr9  | 	at java.base/java.util.Base64$Decoder.decode0(Unknown Source)
solr-lss-dev-solr9  | 
solr-lss-dev-solr9  | 2026-08-13 18:43:12.147 WARN  (qtp549293029-19) [] o.e.j.s.HttpChannel /solr/core-x/select =>org.apache.solr.common.SolrException: Error during request authentication, 
solr-lss-dev-solr9  | 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:354)

I'll poke around and see what I can figure out.

@aelkiss, I fixed this issue. Could you please pull the branch again? It should disappear.

In the Solr 9 log, we expect to see the error. That is something we will handle as part of a separate task, as we will need to update the code.

2026-08-13 15:08:40.338 ERROR (qtp566113173-18-null-237) [c:core-x s:shard1 r:core_node2 x:core-x_shard1_replica_n1 t:null-237] o.a.s.h.RequestHandlerBase Client exception =>org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Query Field 'ctrlnum' is not a valid field name

	at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:235)

aelkiss · 2026-08-13T19:04:52Z

I'm able to run solr 9 and index documents into it, but if I try to query I get no results and this error in the error log:

solr-lss-dev-solr9  | 2026-08-13 18:43:12.146 INFO  (qtp549293029-19-null-66) [ t:null-66] o.a.s.s.SolrDispatchFilter Error authenticating =>java.lang.IllegalArgumentException: Illegal base64 character 5b
solr-lss-dev-solr9  | 	at java.base/java.util.Base64$Decoder.decode0(Unknown Source)
solr-lss-dev-solr9  | 
solr-lss-dev-solr9  | 2026-08-13 18:43:12.147 WARN  (qtp549293029-19) [] o.e.j.s.HttpChannel /solr/core-x/select =>org.apache.solr.common.SolrException: Error during request authentication, 
solr-lss-dev-solr9  | 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:354)

I'll poke around and see what I can figure out.

I had an old commit; the newer one with

PassEnv LSS_SOLR_BASIC_AUTH

should fix that.

With that, querying for * works and gives results. Querying for actual text gives the error about Query Field 'ctrlnum' is not a valid field name.

One other thing I noticed is that the solr 9 container seems to exit after some point; I'm not 100% sure why, but it looks like something is trying to recreate the collection when it already exists:

solr-lss-dev-solr9  | ERROR: 
solr-lss-dev-solr9  | Collection 'core-x' already exists!
solr-lss-dev-solr9  | Checked collection existence using CollectionAdminRequest
solr-lss-dev-solr9  | 
solr-lss-dev-solr9 exited with code 1

We'll need to figure that out soon, but regardless, I think this is good enough as a proof-of-concept for now, and we can work out the rest later.

…olr9 server

…ig can be queried for Solr auth token

… 00-default.conf

liseli requested review from aelkiss, eumalin, kron-spar and moseshll July 9, 2026 20:36

eumalin reviewed Jul 10, 2026

View reviewed changes

moseshll approved these changes Jul 23, 2026

View reviewed changes

eumalin approved these changes Jul 23, 2026

View reviewed changes

moseshll reviewed Aug 12, 2026

View reviewed changes

moseshll reviewed Aug 13, 2026

View reviewed changes

aelkiss reviewed Aug 13, 2026

View reviewed changes

liseli and others added 7 commits August 13, 2026 14:52

Adding solr cloud service to babel

8d254d0

Declare Zookeeper dependency when Solr is extended

3e4fdbd

Update solr.yml to start the Solr server using the image with Solr 9

868ba27

Removed uncommented lines on docker-compose

2820bcf

Set up Solr 6 and Solr 9 in docker environment

d293c9c

add add_basic_auth to send Authorization header to ls has access to S…

6d31165

…olr9 server

Pass Context object $C to __get_request_object so that mdp-conf…

77eba67

…ig can be queried for Solr auth token

liseli added 2 commits August 13, 2026 14:52

adding the context to __get_request_object and LSS_SOLR_BASIC_AUTH to…

2699973

… 00-default.conf

Document Solr credentials in the config files

014ebfc

liseli force-pushed the ETT-1665_babel+fulltextSolr8 branch from 3cf8937 to 014ebfc Compare August 13, 2026 19:55

liseli merged commit 98b521e into main Aug 13, 2026
3 checks passed

liseli deleted the ETT-1665_babel+fulltextSolr8 branch August 13, 2026 20:16

Uh oh!

Conversation

liseli commented Jul 9, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eumalin Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

moseshll commented Jul 14, 2026

Uh oh!

moseshll left a comment

Choose a reason for hiding this comment

Uh oh!

eumalin left a comment

Choose a reason for hiding this comment

Uh oh!

liseli commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eumalin commented Aug 12, 2026

Uh oh!

moseshll commented Aug 12, 2026

Uh oh!

eumalin commented Aug 12, 2026

Uh oh!

moseshll commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moseshll Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

moseshll commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liseli commented Aug 12, 2026

Uh oh!

moseshll Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

aelkiss Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

aelkiss Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

aelkiss commented Aug 13, 2026

Uh oh!

liseli commented Aug 13, 2026

Uh oh!

liseli commented Aug 13, 2026

Uh oh!

aelkiss commented Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liseli commented Jul 9, 2026 •

edited by atlassian Bot

Loading

liseli commented Aug 11, 2026 •

edited

Loading

moseshll commented Aug 12, 2026 •

edited

Loading

moseshll commented Aug 12, 2026 •

edited

Loading