diff --git a/.gitignore b/.gitignore index 51cffeb..fb43a5a 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,6 @@ __pycache__ site/ scripts/generated_tests/ nohup.out -error.log +*.log *.pyc .vscode/settings.json diff --git a/docs/520_2.0.0/sdk/solution_kdp2_host_mipi.md b/docs/520_2.0.0/sdk/solution_kdp2_host_mipi.md index 61b75a4..0f55f10 100644 --- a/docs/520_2.0.0/sdk/solution_kdp2_host_mipi.md +++ b/docs/520_2.0.0/sdk/solution_kdp2_host_mipi.md @@ -27,7 +27,6 @@ User can edit and debug with Keil MDK for further implementation [keil/MDK docs # bin_gen.py will concatenate SCPU/NCPU FW, models_520.nef to generate flash_image.bin # Note that you may need to substitute '/' for '\' in the path ``` -``` - **Program the firmware bin image** Reference: [Jlink programming](../flash_management/flash_management.md#4-program-flash-via-jtagswd-interface) diff --git a/docs/520_2.1.0/sdk/solution_kdp2_host_mipi.md b/docs/520_2.1.0/sdk/solution_kdp2_host_mipi.md index 61b75a4..0f55f10 100644 --- a/docs/520_2.1.0/sdk/solution_kdp2_host_mipi.md +++ b/docs/520_2.1.0/sdk/solution_kdp2_host_mipi.md @@ -27,7 +27,6 @@ User can edit and debug with Keil MDK for further implementation [keil/MDK docs # bin_gen.py will concatenate SCPU/NCPU FW, models_520.nef to generate flash_image.bin # Note that you may need to substitute '/' for '\' in the path ``` -``` - **Program the firmware bin image** Reference: [Jlink programming](../flash_management/flash_management.md#4-program-flash-via-jtagswd-interface) diff --git a/docs/520_2.2.0/sdk/solution_kdp2_host_mipi.md b/docs/520_2.2.0/sdk/solution_kdp2_host_mipi.md index 61b75a4..0f55f10 100644 --- a/docs/520_2.2.0/sdk/solution_kdp2_host_mipi.md +++ b/docs/520_2.2.0/sdk/solution_kdp2_host_mipi.md @@ -27,7 +27,6 @@ User can edit and debug with Keil MDK for further implementation [keil/MDK docs # bin_gen.py will concatenate SCPU/NCPU FW, models_520.nef to generate flash_image.bin # Note that you may need to substitute '/' for '\' in the path ``` -``` - **Program the firmware bin image** Reference: [Jlink programming](../flash_management/flash_management.md#4-program-flash-via-jtagswd-interface) diff --git a/docs/model_training/OpenMMLab/RSN18.md b/docs/model_training/OpenMMLab/RSN18.md index a7a1ba8..15e16b7 100644 --- a/docs/model_training/OpenMMLab/RSN18.md +++ b/docs/model_training/OpenMMLab/RSN18.md @@ -40,7 +40,7 @@ If mmcv and mmcv-full are both installed, there will be `ModuleNotFoundError`. # Step 1: Training models on standard datasets -MMPose provides hundreds of existing and existing pose models in [Model Zoo](https://mmpose.readthedocs.io/en/latest/modelzoo.html), and supports several standard datasets like COCO, MPII, FREIHAND, etc. This note demonstrates how to perform common object detection tasks with these existing models and standard datasets, including: +MMPose provides hundreds of existing and existing pose models in [Model Zoo](https://mmpose.readthedocs.io/en/latest/model_zoo.html), and supports several standard datasets like COCO, MPII, FREIHAND, etc. This note demonstrates how to perform common object detection tasks with these existing models and standard datasets, including: - Use existing models to inference on given images. - Test existing models on standard datasets. diff --git a/docs/model_training/regression.md b/docs/model_training/regression.md index 9df619e..8e5e214 100644 --- a/docs/model_training/regression.md +++ b/docs/model_training/regression.md @@ -82,7 +82,7 @@ lite_hrnet ``` **For MPII data**, please download from [MPII Human Pose Dataset](http://human-pose.mpi-inf.mpg.de/). -The original annotation files have been converted into json format, please download them from [mpii_annotations](https://openmmlab.oss-cn-hangzhou.aliyuncs.com/mmpose/datasets/mpii_annotations.tar). +The original annotation files have been converted into json format, please download them from [mpii_annotations](https://download.openmmlab.com/mmpose/datasets/mpii_annotations.tar). Extract them under `$LITE_HRNET/data`, and make them look like this: ``` diff --git a/docs/toolchain/appendix/command_line.md b/docs/toolchain/appendix/command_line.md index 9f0b6bd..9d90f3a 100644 --- a/docs/toolchain/appendix/command_line.md +++ b/docs/toolchain/appendix/command_line.md @@ -172,20 +172,20 @@ Here is an example JSON with comments. **Please remove all the comments in the r // [optional] // The encryption setting for the batch compiler. Default is not enabled. "encryption": { - // Whether enable encrytion + // Whether enable encryption "whether_encryption": false, - // Encrytion mode selection + // Encryption mode selection // Options: 1, 2 "encryption mode": 1, - // Encrytion key. A hex string. Required in mode 1. + // Encryption key. A hex string. Required in mode 1. "encryption_key": "0x12345678", - // Encrytion file. An absolute path. Required in mode 1. + // Encryption file. An absolute path. Required in mode 1. "key_file": "/data1/enc.txt", - // Encrytion key. A hex string. Required in mode 2, optional in mode 1. + // Encryption key. A hex string. Required in mode 2, optional in mode 1. "encryption_efuse_key": "0x12345678" }, // [optional] - // Whether seperate buffers for each model output. Default is true. + // Whether separate buffers for each model output. Default is true. "dedicated_output_buffer": true, // [optional] // Whether compress weight for saving space. diff --git a/docs/toolchain/appendix/converters.md b/docs/toolchain/appendix/converters.md index aa2e199..0fb03a9 100644 --- a/docs/toolchain/appendix/converters.md +++ b/docs/toolchain/appendix/converters.md @@ -430,7 +430,7 @@ it is the output of a `Gemm`, there is no need to do any more edition. the output, then the model is transposed into channel first. We can use the model editor to safely remove the `Transpose`. -**Step 3:** If the input shape is not availble or invalid, we can use the editor to give it a valid shape. +**Step 3:** If the input shape is not available or invalid, we can use the editor to give it a valid shape. **Step 4:** The model need to pass `onnx2onnx.py` again after running the editor. See [section 6](#6-onnx-to-onnx-onnx-optimization). @@ -469,7 +469,7 @@ Cutting is not recommended to be done with other operations together. positional arguments: in_file input ONNX FILE -out_file ouput ONNX FILE +out_file output ONNX FILE optional arguments: diff --git a/docs/toolchain/appendix/history.md b/docs/toolchain/appendix/history.md index 649f4e9..50fc85c 100644 --- a/docs/toolchain/appendix/history.md +++ b/docs/toolchain/appendix/history.md @@ -28,7 +28,7 @@ * Add Einsum defusion in kneronnxopt. * Support Cast to int64 in knerex and compiler. * Support HardSwish, Topk and Split nodes in knerex and compiler. - * Update the regression flow log printing. Print success log seperately from errors to avoid confusing. + * Update the regression flow log printing. Print success log separately from errors to avoid confusing. * Update IP evaluator for DMA with small length. * Fix the kneronnxopt bug in `replace_Gather_with_Slice`. * Fix the knerex bug: node Concat channel mismatch. @@ -40,7 +40,7 @@ * Add `const_in_bitwidth_mode` option for quantization. The default is int16. Unless the customer particularly desires to increase the speed, it can be changed to int8 * Update analyzer exception log. * Update kneronnxopt to set expanding dilated Conv to False by default. - * Update kneronnxopt to diable fusing BatchNormalization into Conv by default. + * Update kneronnxopt to disable fusing BatchNormalization into Conv by default. * Update compiler for the deep search memory estimation algorithm. * Update compiler to extend the timeout for deep search. * Update compiler to change expt/log/softmax to 16b. @@ -51,7 +51,7 @@ * **[v0.31.0]** * **Introduce `quan_config` for `ModelConfig.analysis` for more detailed quantization configuration.** * **Add `ktc.opt_and_eval` command for quick onnx optimization and evaluation.** - * **Remove deprecated `compilerIpevaluator_730.sh` and add warning messages to other depecated scritps.** + * **Remove deprecated `compilerIpevaluator_730.sh` and add warning messages to other deprecated scripts.** * Add `compiler_tiling` option for IP evaluator. * Add `--clear-shapes` and `--replace-avgpool-with-conv` flags to kneronnxopt. * Add `--seperate` flag to kneronnxopt.onnx_vs_onnx for detailed output comparison. @@ -188,7 +188,7 @@ * Support text procssing models. * Set flatbuffer as the default 720 compiling mode. * Refactor compiler and analyser inner structure. - * **Due to the structure change, batch compiler do not backwark support previous bie files.** + * **Due to the structure change, batch compiler do not backward support previous bie files.** * Refactor toolchain manual. * Bug fixes. * **[v0.19.0]** @@ -198,10 +198,10 @@ * ktc: Add `mode`, `optimize`, `export_dynasty_dump` argument to analysis. * ktc: Set `skip_verify` in analysis as deprecated. * regression: Add optimize option for optimization level selection. - * regression: Fix interface to asure platform is integer. + * regression: Fix interface to assure platform is integer. * converter: Add 720 batch process with `--opt-720` flag. * converter: Add enable shared weight duplication flag `-d`. By default, shared weights are no longer duplicated. - * converter: Remove `-s` flag since it is now the default behavious. + * converter: Remove `-s` flag since it is now the default behavior. * converter: Optimize debug output. * compiler: Fix RDMA not correctly executed. * E2E simulator: Change dynasty library fetching method. diff --git a/docs/toolchain/appendix/nef_utils_guide.md b/docs/toolchain/appendix/nef_utils_guide.md index c7715d7..55fe182 100644 --- a/docs/toolchain/appendix/nef_utils_guide.md +++ b/docs/toolchain/appendix/nef_utils_guide.md @@ -14,8 +14,8 @@ POST_ENC : -E(--enc) -n(--KN) "kn_number"; [Linux Only] options: - -o, --output : output file name prefix, defautl is "model_(target)" - -O, --folder : output folder, defautl is "output/" + -o, --output : output file name prefix, default is "model_(target)" + -O, --folder : output folder, default is "output/" -V, --version : show version number -H, --help : show this message diff --git a/docs/toolchain/appendix/operators.md b/docs/toolchain/appendix/operators.md index 6622cab..6722e9e 100644 --- a/docs/toolchain/appendix/operators.md +++ b/docs/toolchain/appendix/operators.md @@ -246,13 +246,13 @@ Notes: - kernel is kxk & stride is sxs where n and s <= 3 or - kernel_w & stride_w are 1 and kernel_h & stride_h <= 3 or - kernel_h & stride_h are 1 and kernel_w = stride_w <= 3) -26. conditions: not pad in batch && any pad in spacial < 32 && constant mode with 0 const_val +26. conditions: not pad in batch && any pad in spatial < 32 && constant mode with 0 const_val 27. conditions: not pad in batch && any of pad < 32 && constant mode with 0 const_val 28. - 29. conditions: power is 2 30. conditions: keepdims = 1 -31. contitions: keepdims = 1 && reduce not in batch -32. contitions: keepdims = 1 && reduce in ch +31. conditions: keepdims = 1 && reduce not in batch +32. conditions: keepdims = 1 && reduce in ch 33. - 34. conditions: mode != cubic && extrapolation_value is 0 && rank is 4 && phase_init is {0,0} && nearest_mode is floor if mode is nearest && coordinate_transformation_mode != tf_crop_and_resize 35. conditions: mode != cubic && extrapolation_value is 0 && rank is 4 && not both vus_en and hus_en enabled && phase_init_v >= 0 and delta_v <= 1 if vus_en enabled && phase_init_h >= 0 and delta_h <= 1 if hus_en enabled diff --git a/docs/toolchain/appendix/toolchain_webgui.md b/docs/toolchain/appendix/toolchain_webgui.md index eec983a..c030d17 100644 --- a/docs/toolchain/appendix/toolchain_webgui.md +++ b/docs/toolchain/appendix/toolchain_webgui.md @@ -87,7 +87,7 @@ Notes: ## 4. Run IP Evaluator only mode If you only want to evaluate the model to a estimation of the model performance, -you can turn on the `Run Onnx Flow & IP Evaluator only` switch above the preprocess code seciton. +you can turn on the `Run Onnx Flow & IP Evaluator only` switch above the preprocess code section. Then you can press run directly. The evaluation result would be shown on the right side `Console Output` section. ## 5. Stop, restart and remove the service diff --git a/docs/toolchain/appendix/yolo_example.md b/docs/toolchain/appendix/yolo_example.md index 94ad2c4..3add21b 100644 --- a/docs/toolchain/appendix/yolo_example.md +++ b/docs/toolchain/appendix/yolo_example.md @@ -2,7 +2,7 @@ In this document, we provide a step by step example on how to utilize our tools to compile and test with a newly downloaded YOLOv3 model. -> This document is writen for toolchain v0.30.0. If any description is not consistent with the latest toolchain, please refer to the main toolchain manual. +> This document is written for toolchain v0.30.0. If any description is not consistent with the latest toolchain, please refer to the main toolchain manual. ## Tricks for deploying yolo-type detection models and anker based detection models @@ -19,9 +19,9 @@ In this document, we provide a step by step example on how to utilize our tools 2. bbox coordinates with shape 1x4xpixel # at scale 0, 1x4xpixel # at scale 1, ...,1x4xpixel # at scale S 3. Trick: Do NOT concat class scores at different scales. Output class scores for each scale separately. 4. Trick: Do NOT concat class score and coordinates at the same scale. Output class scores and bbox coordinates separately. - 5. Trick: Do NOT concat bbox coordinates at differnt scales. Output bbox coordinates for each scale separately. - 6. Trick: Typically, class scores need to pass activation fuctions such as exp, sigmoid or even softmax. Make sure these activation fucntions are in the model so that quantiztion algorithm can optimize the quantizaiton setting accordingly. - 7. Trick: sometimes, bbox coordinates need to pass exp function or other activation function. Make sure these activation fucntions are in the model so that quantiztion algorithm can optimize the quantizaiton setting accordingly. + 5. Trick: Do NOT concat bbox coordinates at different scales. Output bbox coordinates for each scale separately. + 6. Trick: Typically, class scores need to pass activation functions such as exp, sigmoid or even softmax. Make sure these activation functions are in the model so that quantiztion algorithm can optimize the quantizaiton setting accordingly. + 7. Trick: sometimes, bbox coordinates need to pass exp function or other activation function. Make sure these activation functions are in the model so that quantiztion algorithm can optimize the quantizaiton setting accordingly. 8. Trick: Do NOT concat some outputs and then split in the model. Make sure the computation of all these outputs are separate. If these computation needs to use the same op, the quantization algorithm can detect this situation and share the weights of the same op. @@ -61,7 +61,7 @@ python convert.py yolov3-tiny.cfg yolov3-tiny.weights /data1/yolo.h5 We now have `yolo.h5` under our mounted folder `/data1`. -We also need to preprare some images under the mounted folder. We have provided some example input images at . +We also need to prepare some images under the mounted folder. We have provided some example input images at . Here is how you can get it: @@ -112,7 +112,7 @@ from PIL import Image ## Step 2: IP Evaluation -To make sure the onnx model is as expected, we should check the onnx model's performance and see if there are any unsupprted operators (or CPU nodes). +To make sure the onnx model is as expected, we should check the onnx model's performance and see if there are any unsupported operators (or CPU nodes). ```python # npu (only) performance simulation @@ -145,7 +145,7 @@ The estimated FPS (NPU only) report on your terminal should look similar to this There are two things to take note of in this report: * Found one CPU node 'KneronResize' in our model - Tthe estimated FPS is 22.5861, the report is for NPU only + The estimated FPS is 22.5861, the report is for NPU only At the same time, a folder called `compiler` will be generated in your docker mounted folder (`/data1`); the evaluation result will be found in this folder. One important thing is to check the 'ioinfo.csv' in `/data1/compiler`, which looks like this: diff --git a/docs/toolchain/appendix/yolo_example_InModelPreproc_trick.md b/docs/toolchain/appendix/yolo_example_InModelPreproc_trick.md index 57d3c54..4858726 100644 --- a/docs/toolchain/appendix/yolo_example_InModelPreproc_trick.md +++ b/docs/toolchain/appendix/yolo_example_InModelPreproc_trick.md @@ -6,7 +6,7 @@ In this document, we provide a step by step example on how to utilize our tools > 1. Step 2: Convert and optimize the pretrain model. > 2. Step 4: Check ONNX model and preprocess and postprocess are good. -> This document is writen for toolchain v0.22.0. If any description is not consistent with the latest toolchain, please refer to the main toolchain manual. +> This document is written for toolchain v0.22.0. If any description is not consistent with the latest toolchain, please refer to the main toolchain manual. ## Step 0: Prepare environment and data @@ -44,7 +44,7 @@ python convert.py yolov3-tiny.cfg yolov3-tiny.weights /data1/yolo.h5 We now have `yolo.h5` under our mounted folder `/data1`. -We also need to preprare some images under the mounted folder. We have provided some example input images at . +We also need to prepare some images under the mounted folder. We have provided some example input images at . Here is how you can get it: @@ -106,7 +106,7 @@ m = ktc.onnx_optimizer.onnx2onnx_flow(m) Now, in order to make model porting easier, we do In-Model-Preprocess trick. We add a Batchnormalization layer at model front, this Batchnormalization layer will do the following things: 1. divide 255 for every pixel (data normalization required by this model) -2. add 0.5 (hardware require -128 to 127 input but source data is 0 to 255, so we will substract 128 for source data due to hardware requirement, and add back at model's front) +2. add 0.5 (hardware require -128 to 127 input but source data is 0 to 255, so we will subtract 128 for source data due to hardware requirement, and add back at model's front) ```python @@ -128,7 +128,7 @@ onnx.save(m, 'yolo.opt.onnx') ## Step 3: IP Evaluation -To make sure the onnx model is as expected, we should check the onnx model's performance and see if there are any unsupprted operators (or CPU nodes). +To make sure the onnx model is as expected, we should check the onnx model's performance and see if there are any unsupported operators (or CPU nodes). ```python # npu (only) performance simulation diff --git a/docs/toolchain/manual_1_overview.md b/docs/toolchain/manual_1_overview.md index 7dbcd60..14ed2fd 100644 --- a/docs/toolchain/manual_1_overview.md +++ b/docs/toolchain/manual_1_overview.md @@ -23,7 +23,7 @@ In this document, you'll learn: * Add Einsum defusion in kneronnxopt. * Support Cast to int64 in knerex and compiler. * Support HardSwish, Topk and Split nodes in knerex and compiler. - * Update the regression flow log printing. Print success log seperately from errors to avoid confusing. + * Update the regression flow log printing. Print success log separately from errors to avoid confusing. * Update IP evaluator for DMA with small length. * Fix the kneronnxopt bug in `replace_Gather_with_Slice`. * Fix the knerex bug: node Concat channel mismatch. @@ -36,7 +36,7 @@ In this document, you'll learn: In the following parts of this page, you can go through the basic toolchain working process to get familiar with the toolchain. -Below is a breif diagram showing the workflow of how to generate the binary from a floating-point model using the toolchain. +Below is a brief diagram showing the workflow of how to generate the binary from a floating-point model using the toolchain.
@@ -253,15 +253,15 @@ improve your model performance on-chip waiting in other section. Please check. * [4. Fixed-Point Model Generation](manual_4_bie.md) * [5. Compilation](manual_5_nef.md) -There are also other useful tools and informations: +There are also other useful tools and information: -* [End to End Simulator](appendix/app_flow_manual.md): manual for the E2E simualtor. +* [End to End Simulator](appendix/app_flow_manual.md): manual for the E2E simulator. * [Hardware Performance](appendix/performance.md): common model performance table for Kneron hardwares and the supported operator list. * [Hardware Supported Operators](appendix/operators.md): operators supported by the hardware. * [How to Interpret Fixed-Point Report](appendix/fx_report.md): manual for interpreting the fixed-point report. * [Kneronnxopt](appendix/kneronnxopt.md): manual for the a more flexible onnx optimizer tool. * [ONNX Converters](appendix/converters.md): manual for the script usage of our converter tools. This tool is only for ONNX opset 11/12. -* [Quantization 1 - Introdution to Post-training Quantization](quantization/1.1_Introdution_to_Post-training_Quantization.md): introduction for the quantization. +* [Quantization 1 - Introduction to Post-training Quantization](quantization/1.1_Introduction_to_Post-training_Quantization.md): introduction for the quantization. * [Quantization 2 - Post-training Quantization(PTQ) Flow and Steps](quantization/1.2_Flow_and_Steps.md): manual for the quantization flow and steps. * [Script Tools](appendix/command_line.md): manual for deprecated command line tools. Kept for compatibility of toolchain before v0.15.0) * [Toolchain History](appendix/history.md): manuals for history version of toolchain and the change log. diff --git a/docs/toolchain/manual_2_deploy.md b/docs/toolchain/manual_2_deploy.md index 1558483..2b16525 100644 --- a/docs/toolchain/manual_2_deploy.md +++ b/docs/toolchain/manual_2_deploy.md @@ -93,12 +93,12 @@ their usage: To ensure the quantization tool can work, we recommend the docker has at least 4GB of memory. The actual required size depends on your model size and the image number of quantization. -For Linux uses, by default, docker can share all the CPU and memory resouces of their host machine. So, this isn't a problem. +For Linux uses, by default, docker can share all the CPU and memory resources of their host machine. So, this isn't a problem. But for Windows users, not like Linux, the system resources are not shared. User might want to adjust the resources usage by themselves. -For the docker based on wsl2, as we recommended in the section 1 of this document, it can use update to 50% of your total system memory and all the CPU resources. And here is a artical introduce [how to manage the system resources used by wsl2](https://ryanharrison.co.uk/2021/05/13/wsl2-better-managing-system-resources.html#:~:text=1%20Setting%20a%20WSL2%20Memory%20Limit.%20By%20default,the%20WSL2%20Virtual%20Disk.%20...%204%20Docker.%20). +For the docker based on wsl2, as we recommended in the section 1 of this document, it can use update to 50% of your total system memory and all the CPU resources. And here is a article introduce [how to manage the system resources used by wsl2](https://ryanharrison.co.uk/2021/05/13/wsl2-better-managing-system-resources.html#:~:text=1%20Setting%20a%20WSL2%20Memory%20Limit.%20By%20default,the%20WSL2%20Virtual%20Disk.%20...%204%20Docker.%20). -For the docker based on wsl, users can find the management of the system resouces directly in the setting of the docker. +For the docker based on wsl, users can find the management of the system resources directly in the setting of the docker. For the docker toolbox, it is actually based on the VirtualBox virtual machine. So, user need find which virtual machine the docker is using first. User need to start the docker terminal to ensure the docker is running before we start. And here is following precedue diff --git a/docs/toolchain/manual_3_onnx.md b/docs/toolchain/manual_3_onnx.md index d3b21f4..64e8402 100644 --- a/docs/toolchain/manual_3_onnx.md +++ b/docs/toolchain/manual_3_onnx.md @@ -86,14 +86,14 @@ KL520/KL720/KL530 NPU supports most of the compute extensive OPs, such as Conv, Therefore, we recommend users move certain OPs to preprocess or postprocess. We recommend using the ONNX utils api `onnx.utils.extract_model` for ONNX editing. For details, please refer to the [onnx.utils.extract_model](https://onnx.ai/onnx/api/utils.html). -**Please run the optimzier again after modify the model.** +**Please run the optimizer again after modify the model.** ## 3.2. IP Evaluation Before we start quantizing the model and try simulating the model, we need to test if the model can be taken by the toolchain structure and estimate the performance. IP evaluator is such a tool which can estimate the performance of your model and check if there is any operator or structure not supported by our toolchain. -We need to create a `ktc.ModelConfig` object. The `ktc.ModelConfig` is the class which contains the basic needed information of a model. You can initilize it through the API below. +We need to create a `ktc.ModelConfig` object. The `ktc.ModelConfig` is the class which contains the basic needed information of a model. You can initialize it through the API below. ```python #[API] @@ -160,7 +160,7 @@ classmethod evaluate(output_dir: str = "/data1/kneron_flow") Args: * output_dir (str, optional): output directory. Defaults to "/data1/kneron_flow". -* datapath_bitwidth_mode: choose from "int8"/"int16"/"mix balance"/"mix light". ("int16" is not supported in kdp520. "mix balance" and "mix light" are combines of int8 and int16 mode. "mix balance" prefers int16 while "mix light" prefers int8.) +* datapath_bitwidth_mode: choose from "int8"/"int16"/"mix balance"/"mix light". ("int16" is not supported in kdp520. "mix balance" and "mix light" are combines of int8 and int16 mode. "mix balance" prefers int16 while "mix light" prefers int8. If the model has ReduceL2 operator, we suggest using "mix light" or "int16".) * weight_bitwidth_mode: choose from "int8"/"int16"/"int4"/"mix balance"/"mix light". ("int16" is not supported in kdp520. "int4" is not supported in kdp720. "mix balance" and "mix light" are combines of int8 and int16 mode. "mix balance" prefers int16 while "mix light" prefers int8.) * model_in_bitwidth_mode: choose from "int8"/"int16". ("int16" is not supported in kdp520.) * model_out_bitwidth_mode: choose from "int8"/"int16". ("int16" is not supported in kdp520.) @@ -238,7 +238,7 @@ Since we do not actually has any source model here for the simplicity of example ### 3.4.1. What if the E2E simulator results from the original model and the optimized onnx mismatch? Please double check if the final layers are cut due to unsupported by NPU. -If so, please add the deleted operator as part of the E2E simulater post process and test again. +If so, please add the deleted operator as part of the E2E simulator post process and test again. Otherwise, please search on forum . You can also contact us through the forum if no match issue found. The technical support would reply directly to your post. ### 3.4.2 What if I get "RuntimeError: Inferred shape and existing shape differ in rank: (0) vs (3)'? diff --git a/docs/toolchain/manual_4_bie.md b/docs/toolchain/manual_4_bie.md index 4afa099..7979123 100644 --- a/docs/toolchain/manual_4_bie.md +++ b/docs/toolchain/manual_4_bie.md @@ -27,7 +27,7 @@ Args: * outlier_factor (float, optional): used under 'mmse' mode. The factor applied on outliers. For example, if clamping data is sensitive to your model, set outlier_factor to 2 or higher. Higher outlier_factor will reduce outlier removal by increasing range. Defaults to 1.0. * percentage (float, optional): used under 'percentage' mode. Suggest to set value between 0.999 and 1.0. Use 1.0 for detection models. **Must be smaller than or equal to percentage_16b.** Defaults to 0.999. * percentage_16b (float, optional): used under 'percentage' mode. Suggest to set value between 0.999 and 1.0. Use 1.0 for detection models. **Must be larger than or equal to percentage.** Defaults to 0.999999. -* datapath_bitwidth_mode: choose from "int8"/"int16"/"mix balance"/"mix light"/"mixbw". ("int16" is not supported in kdp520. "mixbw", "mix balance" and "mix light" are combines of int8 and int16 mode. "mix balance" prefers int16 while "mix light" prefers int8. "mixbw" automatically select the best bitwidth for each layer.) +* datapath_bitwidth_mode: choose from "int8"/"int16"/"mix balance"/"mix light"/"mixbw". ("int16" is not supported in kdp520. "mixbw", "mix balance" and "mix light" are combines of int8 and int16 mode. "mix balance" prefers int16 while "mix light" prefers int8. "mixbw" automatically select the best bitwidth for each layer. If the model has ReduceL2 operator, we suggest using "mix light" or "int16".) * weight_bitwidth_mode: choose from "int8"/"int16"/"int4"/"mix balance"/"mix light". ("int16" is not supported in kdp520. "int4" is not supported in kdp720. "mixbw", "mix balance" and "mix light" are combines of int8 and int16 mode. "mix balance" prefers int16 while "mix light" prefers int8. "mixbw" automatically select the best bitwidth for each layer.) * model_in_bitwidth_mode: choose from "int8"/"int16". ("int16" is not supported in kdp520. When "mixbw" is set, this parameter is ignored.) * model_out_bitwidth_mode: choose from "int8"/"int16". ("int16" is not supported in kdp520. When "mixbw" is set, this parameter is ignored.) diff --git a/docs/toolchain/manual_5_nef.md b/docs/toolchain/manual_5_nef.md index 1ce699d..1f9c165 100644 --- a/docs/toolchain/manual_5_nef.md +++ b/docs/toolchain/manual_5_nef.md @@ -20,7 +20,7 @@ Args: * dedicated_output_buffer (bool, optional): dedicated output buffer. Defaults to True. * weight_compress (bool, optional): compress weight to slightly reduce the binary file size. Defaults to False. * hardware_cut_opt (bool, optional): optimize the hardware memory usage while processing large inputs. This option might cause the compiling time increase. Currently, only available for 720. Defaults to False. -* flatbuffer (bool, optional): enable new flatbuffer mode for 720. Defauls to True. +* flatbuffer (bool, optional): enable new flatbuffer mode for 720. Defaults to True. ```python #[API] @@ -40,7 +40,7 @@ Args: * encryption_efuse_key (str, optional): a hex code. Required in mode 2 and optional in mode 1. Defaults to "". * weight_compress (bool, optional): compress weight to slightly reduce the binary file size. Defaults to False. * hardware_cut_opt (bool, optional): optimize the hardware memory usage while processing large inputs. This option might cause the compiling time increase. Currently, only available for 720. Defaults to False. -* flatbuffer (bool, optional): enable new flatbuffer mode for 720. Defauls to True. +* flatbuffer (bool, optional): enable new flatbuffer mode for 720. Defaults to True. We would start with single model first. @@ -102,13 +102,13 @@ Here we use the same input `input_data` which we used in section 3.3. And the `c As mentioned above, we do not provide any postprocess. In reality, you may want to have your own postprocess function in Python, too. -For nef file with mutiple models, we can specify the model with model ID: +For nef file with multiple models, we can specify the model with model ID: ```python hw_results_2 = ktc.kneron_inference(input_data, nef_file=batch_compile_result, input_names=["images"], model_id=32769, platform=720) ``` -After getting the `hw_results` and post-process it, you may want to compare the result with the `fixed_results` which is generated in section 4.2 to see if the results match. If the results mismatch, please contact us direcly through forum . +After getting the `hw_results` and post-process it, you may want to compare the result with the `fixed_results` which is generated in section 4.2 to see if the results match. If the results mismatch, please contact us directly through forum . ## 5.3. NEF Combine (Optional) diff --git a/docs/toolchain/quantization/1.1_Introdution_to_Post-training_Quantization.md b/docs/toolchain/quantization/1.1_Introduction_to_Post-training_Quantization.md similarity index 94% rename from docs/toolchain/quantization/1.1_Introdution_to_Post-training_Quantization.md rename to docs/toolchain/quantization/1.1_Introduction_to_Post-training_Quantization.md index 3365d26..600c61b 100644 --- a/docs/toolchain/quantization/1.1_Introdution_to_Post-training_Quantization.md +++ b/docs/toolchain/quantization/1.1_Introduction_to_Post-training_Quantization.md @@ -1,4 +1,4 @@ -# 1 Introdution to Post-training Quantization +# 1 Introduction to Post-training Quantization Post-training quantization(PTQ) uses a batch of calibration data to calibrate the trained model, and directly converts the trained FP32 model into a fixed-point computing model without any training on the original model. The quantization process can be completed by only adjusting a few hyperparameters, and the process is simple and fast without training. Therefore, this method has been widely used in a large number of device-side and cloud-side deployment scenarios. We recommend that you try the PTQ method to see if it meets the requirements. diff --git a/docs/toolchain/quantization/1.3_Optimizing_Quantization_Modes.md b/docs/toolchain/quantization/1.3_Optimizing_Quantization_Modes.md index 6c0fd2b..026a712 100644 --- a/docs/toolchain/quantization/1.3_Optimizing_Quantization_Modes.md +++ b/docs/toolchain/quantization/1.3_Optimizing_Quantization_Modes.md @@ -17,6 +17,7 @@ This manual explains how to configure quantization bitwidth modes in the Kneron- - Notes: - `"int16"` is not supported on KDP520. - `"mixbw"` (new in Toolchain 0.29.0) automatically selects bitwidth for Conv layers based on quantization sensitivity analysis. + - If the model has ReduceL2 operator, we suggest using `"mix light"` or `"int16"` for `datapath_bitwidth_mode`. 2. `weight_bitwidth_mode`: Controls weight quantization for layers like Conv/Gemm. - Options: `"int8"`, `"int16"`, `"int4"`, `"mix balance"`, `"mix light"`, `"mixbw"` @@ -66,7 +67,7 @@ bie_path = km.analysis( ### 3.2.3 Use `mixbw` for Sensitivity-Guided Quantization -If `mix light` precision is insufficient, use `mixbw`. This mode analyzes Conv node sensitivity and automatically prioritizes 16-bit quantization for sensitive Conv layers. Control compute overhead with flops_ratio (default=0.2). `mixbw` mode may need more time and disk space to evaluate quant sensitivity, but its fps is still faster than all int16. When using `mixbw`, the `model_in_bitwidth_mode`, `model_out_bitwidth_mode`, and `cpu_node_bitwidth_mode` are always `int16` and are not changable. +If `mix light` precision is insufficient, use `mixbw`. This mode analyzes Conv node sensitivity and automatically prioritizes 16-bit quantization for sensitive Conv layers. Control compute overhead with flops_ratio (default=0.2). `mixbw` mode may need more time and disk space to evaluate quant sensitivity, but its fps is still faster than all int16. When using `mixbw`, the `model_in_bitwidth_mode`, `model_out_bitwidth_mode`, and `cpu_node_bitwidth_mode` are always `int16` and are not changeable. ```python diff --git a/mkdocs.yml b/mkdocs.yml index d8535c8..26aeb9d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -346,7 +346,7 @@ nav: - Yolo Example: toolchain/appendix/yolo_example.md - Yolo Example (With In-Model Preprocess): toolchain/appendix/yolo_example_InModelPreproc_trick.md - Quantization: - - 1 Introdution to Post-training Quantization: toolchain/quantization/1.1_Introdution_to_Post-training_Quantization.md + - 1 Introduction to Post-training Quantization: toolchain/quantization/1.1_Introduction_to_Post-training_Quantization.md - 2 Post-training Quantization(PTQ) Flow and Steps: toolchain/quantization/1.2_Flow_and_Steps.md - 3 Optimizing Quantization Modes: toolchain/quantization/1.3_Optimizing_Quantization_Modes.md - Model Training Material: