info.txt

here described basic info about usage and structure of checker

usage and test:
- firstly run "systemctl start docker" to start docker daemon
- you need folowing docker images: python:latest ; ubuntu:latest or/and frolvlad/alpine-gxx:latest
you can choose between alpine and ubuntu by modifying second line of "./config": "ubuntu" / "alpine"
- compile checker: "g++ main.cpp docker.cpp checker.cpp generator.cpp utils.cpp -lcurl -o checker"
- and run checker via: "./checker"
- then you can run "python3 test_service.py ./<path_to_test_json> <type>", where <path_to_test_json>
can be path to one of jsons in tests_json dir or your own and <type> is type of test: "h" (homework) 
or "c" (challenge), some of tests in tests_json are bad and always will result in passed:false
- to view results open "result.json"


tests json:
tests can be sent on /homework and /challenge route
/homework - download files from a given github and can accept different test_type
/challenge - copy files from local source and can test only main test_type

/homework tests:
example:
  "language": "cpp",
  "github_link": "https://github.com/ikoshkila/header_multi_func_test_shtp.git",
  "test_type": "header_test",
  "debug": false,
  "tests_description": [
    { "env": { "mem": 10, "proc": 2, "time": 2 },
      "name": "sum", "submit_id": "101",
      "tests": [ { "input": "100\n120", "output": "220" },
                 { "input": "20\n43",   "output": "63"  },
                 { "input": "7\n123",   "output": "130" } ],
      "types": { "in": ["int","int"], "out": "int" }
    },
    { "env": { "mem": 10, "proc": 2, "time": 2 },
      "name": "str_list", "submit_id": "106",
      "tests": [ { "input": "abcdef",   "output": "a b c d e f "},
                 { "input": "lokira",   "output": "l o k i r a "},
                 { "input": "pgsppl",   "output": "p g s p p l "} ],
      "types": { "in": ["string"], "out": "vector<char>" }
    },
    { "env": { "mem": 10, "proc": 2, "time": 2 },
      "name": "concat", "submit_id": "112",
      "tests": [ { "input": "a\nb",     "output": "ab"    },
                 { "input": "Hi,\nho!", "output": "Hi,ho!"},
                 { "input": "ha\nha",   "output": "haha"  } ],
      "types": { "in": ["string","string"], "out": "string" }
    }
  ]
structure:
  +- language           - programming language
  +- github_link        - link to gihub repo where code to test stored
  +- test_type          - type of test
  +- debug              - set whether to add test results to result.json
  +- test_description   - desctibe tests for each function
     +- name            - name of function
     +- submit_id       - id of test
     +- tests           - tests for current function
     |  +- input        - test input
     |  +- output       - output that function should give
     +- types           - specify input and output types
     |  +- in           - types for input
     |  +- out          - types for output
     +- env             - environment limits
        +- mem          - limit memory usage (swap and regular)
        +- proc         - limit number of pids for test
        +- time         - limit time for test

/challenge tests:
example:
  "language": "cpp",
  "file_path": "/home/admin/Backend/app/challenges/109dj01jdmain.cpp",
  "debug": false,
  "tests_description": {
      "env": { "mem": 10, "proc": 2, "time": 2 },
      "submit_id": "101",
      "tests": [
          { "input": "100\n120", "output": "220" },
          { "input": "20\n43",   "output": "63"  },
          { "input": "7\n123",   "output": "130" }
      ]
  }
structure:
  +- language           - programming language
  +- file_path          - absolute local path to code file
  +- debug              - set whether to add test results to result.json
  +- test_description   - desctibe tests for each function
     +- submit_id       - id of test
     +- tests           - tests for current function
     |  +- input        - test input
     |  +- output       - output that function should give
     +- env             - environment limits
        +- mem          - limit memory usage (swap and regular)
        +- proc         - limit number of pids for test
        +- time         - limit time for test


structure specifications:
  file_path: only absolute local path to .cpp or .py file
  language: for now lanuage can be only "python" or "cpp"
  debug: adds to results debug field, that contain output and expected output
  submit_id: can be any value you like        
  
  test_type description:
  header_test      - test that has only functions and header files  - generate main automaticaly via test json
  main_test        - test that consist only of main file            - not generating main
  header_main_test - test that has everything (header, main, funcs) - not generating main
  
  env description:
  types description:
  types should be specified only if language = cpp or test_type = header_test/header_main_test
  in types should be considered as the types of parameters to be passed to the function
  out types can be types (like return type of function) or array of indixes of in types
  , if test will check for changes of in types (when you pass to in ["string &","string &"]
  and to out [1], then checker compare new value of first in parameter and expected output)
  indixes always must be inited as arrays and regular types as strings
  you can set pointers and addresses to in types, always separate from main type 
  with space ("int *" / "int &")
  in and out types can be set to "void" when test has no input or output
  
  env description:
  name | description                               | units      | default
  time - specify the max time allotted for testing - [seconds]  - 1
  mem  - specify how much memory container can use - [megabyte] - 6
  proc - specify max number of processes           -            - 2


result json:
example:
  [{"error":0,"solved":true,"submit_id":"1","tests_results":[
    {"OOM":false,"duration":"0:0:0.001244","exitcode":"0","passed":true},
    {"OOM":false,"duration":"0:0:0.000367","exitcode":"0","passed":true} ]},
  {"error":0,"solved":true,"submit_id":"2","tests_results":[
    {"OOM":false,"duration":"0:0:0.001244","exitcode":"0","passed":true} ]}
  ]
structure:
  +- error         - internal error code
  +- solved        - does whole test solved
  +- submit_id     - id of test
  +- test_results  - results of each tests
  |  +- OOM        - does out of memory error occurred
  |  +- duration   - test execution time
  |  +- exitcode   - ubuntu or alpine exit codes for test process
  |  +- passed     - does this individual test passed
  +- error_msg     - describw internal error if error more than 0
structure specifications:
  error:
  0 - no error
  1 - compilation error
  2 - the main file already exist (only for header_test)
  3 - github repository is private or does not exist
  4 - cant load config
  5 - nothing, just something went wrong
  6 - cant find any files to test

  error_msg: if internal error occurre, than error_msg field appears and give more info
  submit_id: submit id passed at test json
  OOM: out of memory error occurred while test runing
  
  exitcode:
  all fatal error signals for ubuntu / alpine / python
  for cpp:
    if OOM/timeout/pids-limit reached or occurred, then exitcode should be 9 (sigkill exitcode)
    125 - if docker run command fails
    126 - if container command connot be invoked
    127 - if container command connot be found
  for python:
    1 - if error while runing (traceback)
    2 - cannot find file


other:
  config:
  config just store last time when workspace dir was cleared and what container
  image should be used for cpp compilation and test, can be alpine or ubuntu
  
  all tests checked by comapring output of code gived for test and expected output
  output specific:
  - output for vectors and must end with " " and list all elements in one line, so for example
  we need to print vector [1,2,3,4,5,6,7] then output need to be "1 2 3 4 5 6 7 "
  - output for floats must be with 5 numbers after deciaml, so float 145.15 -> 145.15000, 0.0 -> 0.00000

  all tests are checks by comparing the code output and expected output
  output specifications:
  - output for vectors should end with " " and list all elements in one line, e.g.
  we need to output the vector [1,2,3,4,5,6,7], then the output should be "1 2 3 4 5 6 7 "
  - output for floating numbers should always has 5 numbers after decimal point,
  so float 145.15 -> 145.15000 or 0.0 -> 0.00000