Skip to content

feat: honor --no-log for history and upload in parallel (AI生成) - #108

Merged
pluveto merged 2 commits into
mainfrom
feat/issue-89-history-and-parallel
Sep 1, 2026
Merged

pluveto merged 2 commits into
mainfrom
feat/issue-89-history-and-parallel

Conversation

@pluveto

@pluveto pluveto commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Closes #89

--no-log now disables both upgit.log and history.log. Batch GitHub (and other) uploads run with at most 8 in-flight upload messages, results stay in input order, and the first error fails the run.

What changed

  • History is an object: files writes both logs, silent discards record. App constructs it once from --no-log; it does not branch around two free functions.
  • Uploader: Send + Sync and Box<dyn Uploader + Send + Sync>. The message is still upload(&self, artifact, key).
  • BatchPublisher fans that message out with a bound of 8 (std threads, no rayon). Single-file runs stay on the calling thread.
  • Help + README / docs one-liners match the flag.

How to verify

cargo test --workspace --locked
cargo clippy --workspace --locked --all-targets -- -D warnings
  • silent_writes_no_files — History::silent creates neither log
  • files_writes_history_and_upload_log — both files are written when logging is on
  • batch_publisher_sends_one_upload_per_artifact_and_keeps_order — 12 concurrent messages, URLs in input order
  • batch_publisher_fails_the_run_on_first_error — error fails the run; remaining files are not all started

Manual: upgit -n --application-path /tmp/upgit-nolog a.svg b.svg must not create history.log or upgit.log there. A batch of many small GitHub Contents uploads should finish far under the previous serial ~3 min / 183 files, and stdout order must match the file list (Typora).

Three giants (deliberate)

Eric Evans:
--no-log is the ubiquitous language for “this run does not journal.” history.log is a log; leaving it on splits the language from what the user and the flag name already mean. Do not add --no-history. History is an application side effect of a successful publish, not part of the Uploader context. Parallelism is not a GitHub/S3/recipe concept; the uploader still puts one artifact at one key. The run fails as one outcome — do not invent a “partial success” aggregate.

John Ousterhout:
Design twice. (1) if !no_log around record_history / record_upload_log, plus Uploader::upload_batch or rayon over 183 files. (2) One History object (write vs silent) and a BatchPublisher that hides a bounded thread pool. (1) is shallow: App keeps two log paths and the uploader grows a second message. (2) hides files, threads, and fail-fast. Done means: --no-log creates neither file; at most 8 in flight; stdout order matches input; first error (lowest index) fails; no new deps. Journal after a successful batch so tiny appends cannot serialize HTTP.

Alan Kay:
Objects and messages. History answers record. Silent vs files is chosen at construction — App does not orchestrate two functions. Uploader still answers one upload message; Send + Sync only so several threads can deliver it. BatchPublisher is the sender that fans those messages out. Do not put upload_batch on the uploader. Do not make App a thread scheduler with closures over free functions.

三个人的讨论

Kay: workers should record as each upload completes — live objects collaborating, closer to the old serial “history of what went up.”

Ousterhout: that puts a mutex and two opens on the upload path. The issue asked for no extra disk bottleneck. Emitter already waits for the full ordered set; History is the same after-the-fact journal. On failure, some HTTP PUTs may already have happened — that is inherent to parallel I/O, not a reason to interleave logging.

Evans: the application “upload run” is the consistency boundary for output. Partial journals are not a domain invariant we promised. --no-log covering both files is the language fix; a second flag would be a worse split.

Ousterhout: then BatchPublisher is acceptable only if Uploader stays one-artifact and concurrency is bounded. Reject rayon, reject 183 threads, reject skipping failed files and printing the rest.

Kay agrees: the extra object is the sender, not a second upload message. Concedes recording after the batch.

Evans: no Go 0.2 forward-compat; the old “history.log is still written” help text was the bug.

收敛结论

已收敛. --no-log → History::silent() (neither file). Otherwise History::files writes both, in input order, after a successful batch. BatchPublisher sends upload with 8 workers via thread::scope; 0–1 artifacts stay serial. Uploader: Send + Sync. First error fails the run; in-flight work finishes; further artifacts are not started. Output order preserved.

Rejected:

  • Second flag --no-history — splits ubiquitous language
  • Uploader::upload_batch — parallelism is the sender’s concern
  • Rayon / unbounded spawn — extra dep or 183 threads
  • App if no_log around two free functions — shallow; two log paths leak
  • Continuing after errors and omitting failed files — silent skip

决策与后续

Decided:

  • One flag, both logs
  • History object, record after successful batch
  • Bound 8, std threads, ordered results, fail on first error
  • GitHubUploader unchanged except Send+Sync auto-traits

Residual risk: a failed batch may still have uploaded some in-flight files (same class of leftover as serial-stop-at-N). GitHub secondary rate limits under 8 concurrent Contents PUTs are possible; 8 is conservative versus 183 serial.

`--no-log` now silences both history.log and upgit.log through a History
object. BatchPublisher sends one upload message per artifact with at most
eight in flight and keeps output order.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

注意相当多的uploader实际上禁止并发,例如 Github。剩余的也都有限流。默认应该不并发,用户显式指定才能并;发。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43e13dc. Default concurrency is 1 (serial). Parallel only if the user passes --jobs N. GitHub Contents stays serial unless they opt in.

Comment thread crates/upgit-core/src/publisher.rs Outdated
Comment on lines +131 to +158
fn collect_slots(slots: Vec<Slot>) -> Result<Vec<(Locator, PublicUrl)>, PublishError> {
let n = slots.len();
let mut out = Vec::with_capacity(n);
let mut first_err = None;
let mut missing = false;
for slot in slots {
match slot.into_inner().expect("slot") {
Some(Ok(value)) => {
if first_err.is_none() {
out.push(value);
}
}
Some(Err(err)) => {
if first_err.is_none() {
first_err = Some(err);
}
}
None => missing = true,
}
}
if let Some(err) = first_err {
return Err(err);
}
if missing {
return Err(UploadError::message("upload did not finish").into());
}
Ok(out)
}

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expect 消息不规范。且此函数不应该放到 top level

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43e13dc. collect_slots is a private associated function on BatchPublisher. Mutex poison uses into_inner().unwrap_or_else(|e| e.into_inner()) instead of expect("slot").


/// `raw` is the locator before `[link]` replacements; `url` is after them.
/// `shown` is what the user sees (raw or rewritten, depending on `--raw`).
pub fn record(&self, raw: &str, url: &str, key: &str, shown: &str) -> io::Result<()> {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

此处公共函数注释不规范

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43e13dc. Public rustdoc on History / record is a one-line description of the journal; no --no-log construction notes.

Comment thread crates/upgit/src/history.rs Outdated
/// Append one JSON history line: `{"time":"...","rawUrl":"...","url":"..."}`.
fn append_history(path: &Path, raw: &str, url: &str) -> io::Result<()> {
ensure_parent(path)?;
let line = serde_json::json!({

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

避免使用free JSON

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43e13dc. Replaced serde_json::json! with HistoryLine { time, raw_url, url } (rawUrl via serde rename).

Comment thread crates/upgit/src/history.rs Outdated
}

/// Append one JSON history line: `{"time":"...","rawUrl":"...","url":"..."}`.
fn append_history(path: &Path, raw: &str, url: &str) -> io::Result<()> {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

避免使用top level无主函数

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43e13dc. append_history / append_upload_log / ensure_parent / timestamp are private methods on FileJournal.

Comment thread docs/README.zh-CN.md Outdated
6. 可执行文件同目录下的 `config.toml` 和 `upgit.toml`

`--application-path` 会改变可执行文件同目录的查找位置。每次上传后会在该目录写入 `history.log`(除非 `--no-log`,否则还有 `upgit.log`)。
`--application-path` 会改变可执行文件同目录的查找位置。每次上传后会在该目录写入 `history.log` 和 `upgit.log`(`--no-log` 时两者都不写)。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不要用这种虚拟指代,应该写清楚是什么

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 43e13dc. The zh line now names history.log and upgit.log (no 「两者」). EN/zh both say uploads are serial unless --jobs is set.

GitHub Contents and other hosts rate-limit concurrent PUTs, so
BatchPublisher defaults to 1. Pass --jobs N to opt in. History helpers
live on FileJournal; history lines use a Serialize struct; collect_slots
is a BatchPublisher method and recovers mutex poison without expect.
@pluveto
pluveto merged commit 01cd434 into main Sep 1, 2026
3 checks passed
@pluveto
pluveto deleted the feat/issue-89-history-and-parallel branch September 1, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant