NRL-1875 Extract size must be a mulitple of buffer size, else will round up to the next multiple e.g. 1.5m -> 2m

Author: anjalitrace2-nhs

Makefile

@@ -356,7 +356,7 @@ perftest-generate-pointer-table-extract:
 	@echo "Generating pointer table extract with PERFTEST_TABLE_NAME=$(PERFTEST_TABLE_NAME) and ENV=$(ENV) and DIST_PATH=$(DIST_PATH)"
 	rm -rf "${DIST_PATH}/nft"
 	mkdir -p "${DIST_PATH}/nft"
-	PYTHONPATH=. poetry run python tests/performance/perftest_environment.py generate_pointer_table_extract --output_dir="${DIST_PATH}/nft" --extract-size=1500000
+	PYTHONPATH=. poetry run python tests/performance/perftest_environment.py generate_pointer_table_extract --output_dir="${DIST_PATH}/nft" --extract-size=2000000
 	./scripts/get-current-info.sh > "${DIST_PATH}/nft/info.json"
 	zip -r "${DIST_PATH}/pointer_extract-${PERFTEST_TABLE_NAME}.zip" "${DIST_PATH}/nft"
 	aws s3 cp "${DIST_PATH}/pointer_extract-${PERFTEST_TABLE_NAME}.zip" "s3://nhsd-nrlf--${ENV}-metadata/performance/seed-pointers-extract-${PERFTEST_TABLE_NAME}.zip"

scripts/seed_nft_tables.py

@@ -111,7 +111,6 @@ def pointer_is_processed(pointer):
             for unprocessed_item in unprocessed_items
             if unprocessed_item["PutRequest"]["Item"].get("id") == pointer_id
         ]
-        # print("unprocessed matches:", matches)
 
         return len(matches) == 0
 

tests/performance/README.md

@@ -113,7 +113,7 @@ make perftest-generate-pointer-table-extract \
    PERFTEST_TABLE_NAME=nhsd-nrlf--perftest-anjali-test-2-pointers-table ENV=perftest
 ```
 
-This will generate a csv extract of the given pointer table containing a row per pointer. To run the perf tests, you will need an extract larger than the number of test iterations. The default extract size is 1.5 million - this can be changed in the make file command by updating the value of`--extract-size`. Too big and the test runners will take a long time to load the file.
+This will generate a csv extract of the given pointer table containing a row per pointer. To run the perf tests, you will need an extract larger than the number of test iterations. The default extract size is 2 million - this can be changed in the make file command by updating the value of`--extract-size`. Too big and the test runners will take a long time to load the file.
 
 ## Assumptions / Caveats
 

tests/performance/perftest_environment.py

@@ -66,7 +66,7 @@ def __next__(self):
 
 
 def generate_pointer_table_extract(
-    extract_size=1500000,
+    extract_size=2000000,  # must be a multiple of buffer_size (1 million)
     output_dir=".",
 ):
     """