Restructured package + Improved guess mode

Author: dhondta

docs/pages/enc/compressions.md

@@ -94,7 +94,7 @@ This implements multiple compression types available in the native [`zipfile`](h
 
 ### Run-Length Encoding
 
-This replaces each run of identical consecutive characters by its length followed by the character. The default variant writes the length in decimal ; it cannot encode digits, as they would be confused with the lengths. The `compact` variant omits a length of 1, the `suffix` variant writes the length after the character (both can be combined as `rle-compact-suffix`), while the `bytes` variant writes the length as a single character (code point 1 to 255) and can encode any input. The `escape` variant keeps single characters as they are and writes a run as the character twice followed by the number of extra repeats as a single character (code point 0 to 255) ; it can encode any input and does not expand text without repeats.
+This replaces each run of identical consecutive characters by its length followed by the character. The default variant writes the length in decimal ; it cannot encode digits, as they would be confused with the lengths. The `compact` variant omits a length of 1, the `suffix` variant writes the length after the character (both can be combined as `rle-compact-suffix`), while the `bytes` variant writes the length as a single character (code point 1 to 255) and can encode any input. The `escape` variant keeps single characters as they are and writes a run as the character twice followed by the number of extra repeats as a single character (code point 0 to 255) ; it can encode any input and does not expand text without repeats. When decoding the textual variants, an output longer than 2^26 characters is handled as an error (e.g. while guessing on random digits).
 
 **Codec** | **Conversions** | **Aliases** | **Comment**
 :---: | :---: | --- | ---

docs/pages/guessing.md

@@ -14,6 +14,11 @@ The following parameters are tunable:
 - `include`: the [categories](features.md#list-codecs), codecs or encodings to be considered, as a single name, a list of names or a dictionary of lists of names per depth (from 0, -1 meaning any depth) ; by default, `None`, meaning all of them (very slow).
 - `exclude`: the [categories](features.md#list-codecs), codecs or encodings not to be considered, in the same formats ; by default, `None`, meaning none of them.
 - `found`: a list or tuple of currently found encodings that can be used to save time if the first decoding steps are known ; by default, an empty tuple.
+- `timeout`: the maximum duration of the search, in seconds ; by default, `None`, meaning no limit.
+- `max_nodes`: the maximum number of expanded candidates, each expansion decoding a candidate with every considered encoding ; by default, `None`, meaning no limit.
+- `beam`: the number of best-ranked decodings of each candidate that are kept for the next depth ; by default, `None`, meaning all of them.
+
+When the `timeout` or `max_nodes` bound is reached, the search stops and returns the results found so far. The search decodes and ranks each candidate only once, level by level, and expands a decoded output only once even when several chains of encodings lead to it (the best-ranked chain is kept).
 
 The result is a dictionary of the decoded strings by tuple of encodings, in order of application.
 
@@ -81,7 +86,7 @@ Another example of 2-stages encoded string:
 {('base64', 'morse'): 'this is a test'}
 ```
 
-With the [CLI tool](cli.md), these parameters are given with the `-i`/`--include` and `-e`/`--exclude` options, the names being possibly prefixed with a depth (e.g. `codext guess -i 0:base62 1:base64`).
+With the [CLI tool](cli.md), these parameters are given with the `-i`/`--include` and `-e`/`--exclude` options, the names being possibly prefixed with a depth (e.g. `codext guess -i 0:base62 1:base64`), while the bounds of the search are given with the `-t`/`--timeout`, `-n`/`--max-nodes` and `-b`/`--beam` options.
 
 When multiple results are expected, `stop` and `show` arguments can be used respectively to avoid stopping while finding a result and to display the intermediate result.
 
@@ -98,7 +103,7 @@ A few stop functions are predefined in the `stopfunc` submodule.
 ```python
 >>> import codext
 >>> dir(codext.stopfunc)
-['LANG_BACKEND', 'LANG_BACKENDS', ..., '_reload_lang', 'default', 'flag', ..., 'printables', 'regex', 'text']
+['LANG_BACKEND', 'LANG_BACKENDS', ..., '_reload_lang', 'default', 'flag', ..., 'printables', 'regex', 'text', 'text_strict']
 ```
 
 Currently, the following stop functions are provided:
@@ -108,6 +113,7 @@ Currently, the following stop functions are provided:
 - `printables`:     checks that every output character is in the set of printables
 - `regex(pattern)`: takes one argument, the regular expression, for checking a string against the given pattern
 - `text`:           checks for printables and an entropy less than 4.6 (empirically determined)
+- `text_strict`:    checks for `text` and a [textness](#ranking-heuristic) of at least 0.5, hence rejects encoded-looking strings (e.g. Base64) that `text` accepts ; as the search then goes deeper, it is best used with a bound (e.g. `timeout=60`), while flags are better found with `flag`
 
 A stop function can be used as the second argument of the `guess` function or as a keyword-argument, as shown in the following examples:
 
@@ -173,13 +179,13 @@ Each time a backend is loaded, it gets `lang_**` stop functions attached to the
 
 ### Ranking Heuristic
 
-At each step of the search, the candidate encodings are decoded and ranked with a score, the best ones being tried first. The score combines characteristics of the input compared to those expected by each codec (length of its charset, padding, rate of printable characters, expansion factor and entropy) and a characteristic of the decoded output: how much it looks like a text.
+At each step of the search, the candidate encodings are decoded and ranked with a score, the best ones being tried first. The score combines characteristics of the input compared to those expected by each codec (length of its charset, padding, whole quanta for the bases whose N is a power of 2, e.g. 4 characters for Base64, rate of printable characters, expansion factor and entropy, the padding being left out of these characteristics) and a characteristic of the decoded output: how much it looks like a text.
 
-This "textness" depends on the script of the letters. For Latin letters, it is mainly given by the rate of common bigrams among the pairs of successive letters (around 70% for English texts, against around 15% for random letters ; most of these bigrams are also common in other Latin-script languages such as French, German, Spanish or Italian), weighted by the rates of printable characters and of letters and spaces, and by the number of distinct letters. For other scripts (e.g. Cyrillic, Greek, Arabic or CJK), it only relies on language-independent features: the rate of letters of the dominant script, the rate of letters, spaces and punctuation, and the length of the words (for the scripts that separate words with spaces). As these features cannot distinguish a text from a substitution of its characters, the outputs of such codecs are only rated from Latin letters. A bonus is given when the output looks like a text (textness above 0.5) and more than the input, which favours the right decoding over transformations that never fail (e.g. keyboard shifts) without favouring wrong decodings in a chain of encodings. Codecs that only substitute characters (e.g. `rot`, `xor`, `kbshift`) also get a penalty when their output looks less like a text than their input.
+This "textness" depends on the script of the letters. For Latin letters, it is mainly given by the rate of common bigrams among the pairs of successive letters (around 70% for English texts, against around 15% for random letters ; most of these bigrams are also common in other Latin-script languages such as French, German, Spanish or Italian), weighted by the rate of printable characters, the rate of letters and spaces (at least 60% in a text), the number of distinct letters, the length of the words, the rate of tokens that are words or numbers (unlike e.g. a keyboard shift of an encoded string) and the function words such as "the", "et" or "und" (a reversed text, which holds the same bigrams, holds reversed function words such as "eht" instead). For other scripts (e.g. Cyrillic, Greek, Arabic or CJK), it only relies on language-independent features: the rate of letters of the dominant script, the rate of letters, spaces and punctuation, and the length of the words (for the scripts that separate words with spaces). As these features cannot distinguish a text from a substitution of its characters, the outputs of such codecs are only rated from Latin letters. A bonus is given when the output looks like a text (textness above 0.5) and more than the input, which favours the right decoding over transformations that never fail (e.g. keyboard shifts) without favouring wrong decodings in a chain of encodings. Codecs that only substitute characters (e.g. `rot`, `xor`, `kbshift`) also get a penalty when their output looks less like a text than their input.
 
 !!! warning "Work in progress"
 
-    This part shall still be improved with better features and/or using machine learning. In particular, the default stop function (`text`) accepts any printable output with a low entropy, hence possibly an intermediate result of a chain of encodings.
+    This part shall still be improved with better features and/or using machine learning. In particular, the default stop function (`text`) accepts any printable output with a low entropy, hence possibly an intermediate result of a chain of encodings ; `text_strict` avoids this at the cost of a deeper, hence longer, search.
 
 ### Strength Evaluation
 

docs/pages/howto.md

@@ -2,8 +2,8 @@ The purpose of this section is to provide a tutorial for creating new codecs acc
 
 As explained in [this section](features.md), `codext` provides the possibility to add new codecs in two ways:
 
-1. [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L56): using this function, the *encode* and *decode* functions must be given as arguments.
-2. [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160): using this function, an *encoding map* must be given but can be formatted in different ways to handle various use cases.
+1. [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L147): using this function, the *encode* and *decode* functions must be given as arguments.
+2. [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322): using this function, an *encoding map* must be given but can be formatted in different ways to handle various use cases.
 
 In both cases, a *pattern* is given in argument and aims to define the set of all strings that aim to select this codec.
 
@@ -49,29 +49,29 @@ Whatever solution is chosen, the following arguments shall be considered:
 
 ### Which `add` function ?
 
-At this point, it is necessary to determine what kind of codec you want. If it is a simple map of characters, you should definitely use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160). If it is more complex and cannot be handled using [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160)'s options, then you should use [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L56) and define the encode/decode functions yourself.
+At this point, it is necessary to determine what kind of codec you want. If it is a simple map of characters, you should definitely use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322). If it is more complex and cannot be handled using [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322)'s options, then you should use [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L147) and define the encode/decode functions yourself.
 
 A few examples:
 
-- `morse` is a simple map that does not handle case ; it then uses [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160) with `ignore_case` set to "`encode`" (not "`both`" for encoding and decoding as it does not matter anyway for decoding)
-- `whitespace` has 2 codecs defined ; the simple one is a simple bit encoding map, therefore using [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160) with `intype` set to "`bin`" (for pre-converting characters to bits before applying the encoding map), and the complex one uses [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L56) with its specific endocde/decode functions
+- `morse` is a simple map that does not handle case ; it then uses [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322) with `ignore_case` set to "`encode`" (not "`both`" for encoding and decoding as it does not matter anyway for decoding)
+- `whitespace` has 2 codecs defined ; the simple one is a simple bit encoding map, therefore using [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322) with `intype` set to "`bin`" (for pre-converting characters to bits before applying the encoding map), and the complex one uses [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L147) with its specific endocde/decode functions
 - `atbash` defines a dynamic map with a "factory" function, that creates the encoding map according to the parameters supplied in the codec name
 
 So, before going further, determine the following:
 
-- What does the new codec map from and to ? E.g. if binary input and ordinal output, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160) with `intype="bin"` and `outype="ord"`.
-- Is this codec ignoring case ? If so, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160) and specify which operation(s) should ignore case, e.g. `ignore_case="both"` or `ignore_case="decode"`.
-- Should this codec handle no error ? If so, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160) do not forget to specify `no_error=True`.
-- Does the codec yields variable-length encoded tokens ? If so, you can still use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160) but you should define `sep` (separator) as `codext` will not be able to handle ambiguities.
+- What does the new codec map from and to ? E.g. if binary input and ordinal output, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322) with `intype="bin"` and `outype="ord"`.
+- Is this codec ignoring case ? If so, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322) and specify which operation(s) should ignore case, e.g. `ignore_case="both"` or `ignore_case="decode"`.
+- Should this codec handle no error ? If so, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322) do not forget to specify `no_error=True`.
+- Does the codec yields variable-length encoded tokens ? If so, you can still use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322) but you should define `sep` (separator) as `codext` will not be able to handle ambiguities.
 
-If you find aspects that are not covered in these questions, you shall use [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L56), then refering to [Case 1](#case-1-generic-encoding-definition). Otherwise, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160) and refer 
+If you find aspects that are not covered in these questions, you shall use [`add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L147), then refering to [Case 1](#case-1-generic-encoding-definition). Otherwise, you can use [`add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322) and refer 
 to [Case 2](#case-2-encoding-map).
 
 -----
 
 ### Case 1: Generic encoding definition
 
-This uses: [`codext.add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L56)
+This uses: [`codext.add`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L147)
 
 This applies when the codec is more complex than a mapping, as defined in _[Case 2: Encoding map](#case-2-encoding-map)_.
 
@@ -129,7 +129,7 @@ _handle_error(token, position)
 
 ### Case 2: Encoding map
 
-This uses: [`codext.add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L160)
+This uses: [`codext.add_map`](https://github.com/dhondta/python-codext/blob/main/src/codext/__common__.py#L322)
 
 This applies when the codec can be defined a simple mapping between source and destination tokens.