Repository navigation
feat: honor --no-log for history and upload in parallel (AI生成) - #108
Conversation
`--no-log` now silences both history.log and upgit.log through a History object. BatchPublisher sends one upload message per artifact with at most eight in flight and keeps output order.
There was a problem hiding this comment.
注意相当多的uploader实际上禁止并发,例如 Github。剩余的也都有限流。默认应该不并发,用户显式指定才能并;发。
There was a problem hiding this comment.
Fixed in 43e13dc. Default concurrency is 1 (serial). Parallel only if the user passes --jobs N. GitHub Contents stays serial unless they opt in.
| fn collect_slots(slots: Vec<Slot>) -> Result<Vec<(Locator, PublicUrl)>, PublishError> { | ||
| let n = slots.len(); | ||
| let mut out = Vec::with_capacity(n); | ||
| let mut first_err = None; | ||
| let mut missing = false; | ||
| for slot in slots { | ||
| match slot.into_inner().expect("slot") { | ||
| Some(Ok(value)) => { | ||
| if first_err.is_none() { | ||
| out.push(value); | ||
| } | ||
| } | ||
| Some(Err(err)) => { | ||
| if first_err.is_none() { | ||
| first_err = Some(err); | ||
| } | ||
| } | ||
| None => missing = true, | ||
| } | ||
| } | ||
| if let Some(err) = first_err { | ||
| return Err(err); | ||
| } | ||
| if missing { | ||
| return Err(UploadError::message("upload did not finish").into()); | ||
| } | ||
| Ok(out) | ||
| } |
There was a problem hiding this comment.
expect 消息不规范。且此函数不应该放到 top level
There was a problem hiding this comment.
Fixed in 43e13dc. collect_slots is a private associated function on BatchPublisher. Mutex poison uses into_inner().unwrap_or_else(|e| e.into_inner()) instead of expect("slot").
|
|
||
| /// `raw` is the locator before `[link]` replacements; `url` is after them. | ||
| /// `shown` is what the user sees (raw or rewritten, depending on `--raw`). | ||
| pub fn record(&self, raw: &str, url: &str, key: &str, shown: &str) -> io::Result<()> { |
There was a problem hiding this comment.
Fixed in 43e13dc. Public rustdoc on History / record is a one-line description of the journal; no --no-log construction notes.
| /// Append one JSON history line: `{"time":"...","rawUrl":"...","url":"..."}`. | ||
| fn append_history(path: &Path, raw: &str, url: &str) -> io::Result<()> { | ||
| ensure_parent(path)?; | ||
| let line = serde_json::json!({ |
There was a problem hiding this comment.
Fixed in 43e13dc. Replaced serde_json::json! with HistoryLine { time, raw_url, url } (rawUrl via serde rename).
| } | ||
|
|
||
| /// Append one JSON history line: `{"time":"...","rawUrl":"...","url":"..."}`. | ||
| fn append_history(path: &Path, raw: &str, url: &str) -> io::Result<()> { |
There was a problem hiding this comment.
Fixed in 43e13dc. append_history / append_upload_log / ensure_parent / timestamp are private methods on FileJournal.
| 6. 可执行文件同目录下的 `config.toml` 和 `upgit.toml` | ||
|
|
||
| `--application-path` 会改变可执行文件同目录的查找位置。每次上传后会在该目录写入 `history.log`(除非 `--no-log`,否则还有 `upgit.log`)。 | ||
| `--application-path` 会改变可执行文件同目录的查找位置。每次上传后会在该目录写入 `history.log` 和 `upgit.log`(`--no-log` 时两者都不写)。 |
There was a problem hiding this comment.
Fixed in 43e13dc. The zh line now names history.log and upgit.log (no 「两者」). EN/zh both say uploads are serial unless --jobs is set.
GitHub Contents and other hosts rate-limit concurrent PUTs, so BatchPublisher defaults to 1. Pass --jobs N to opt in. History helpers live on FileJournal; history lines use a Serialize struct; collect_slots is a BatchPublisher method and recovers mutex poison without expect.
Closes #89
--no-lognow disables bothupgit.logandhistory.log. Batch GitHub (and other) uploads run with at most 8 in-flightuploadmessages, results stay in input order, and the first error fails the run.What changed
Historyis an object:fileswrites both logs,silentdiscardsrecord. App constructs it once from--no-log; it does not branch around two free functions.Uploader: Send + SyncandBox<dyn Uploader + Send + Sync>. The message is stillupload(&self, artifact, key).BatchPublisherfans that message out with a bound of 8 (std threads, no rayon). Single-file runs stay on the calling thread.How to verify
cargo test --workspace --locked cargo clippy --workspace --locked --all-targets -- -D warningssilent_writes_no_files—History::silentcreates neither logfiles_writes_history_and_upload_log— both files are written when logging is onbatch_publisher_sends_one_upload_per_artifact_and_keeps_order— 12 concurrent messages, URLs in input orderbatch_publisher_fails_the_run_on_first_error— error fails the run; remaining files are not all startedManual:
upgit -n --application-path /tmp/upgit-nolog a.svg b.svgmust not createhistory.logorupgit.logthere. A batch of many small GitHub Contents uploads should finish far under the previous serial ~3 min / 183 files, and stdout order must match the file list (Typora).Three giants (deliberate)
Eric Evans:
--no-logis the ubiquitous language for “this run does not journal.”history.logis a log; leaving it on splits the language from what the user and the flag name already mean. Do not add--no-history. History is an application side effect of a successful publish, not part of the Uploader context. Parallelism is not a GitHub/S3/recipe concept; the uploader still puts one artifact at one key. The run fails as one outcome — do not invent a “partial success” aggregate.John Ousterhout:
Design twice. (1)
if !no_logaroundrecord_history/record_upload_log, plusUploader::upload_batchor rayon over 183 files. (2) One History object (write vs silent) and a BatchPublisher that hides a bounded thread pool. (1) is shallow: App keeps two log paths and the uploader grows a second message. (2) hides files, threads, and fail-fast. Done means:--no-logcreates neither file; at most 8 in flight; stdout order matches input; first error (lowest index) fails; no new deps. Journal after a successful batch so tiny appends cannot serialize HTTP.Alan Kay:
Objects and messages.
Historyanswersrecord. Silent vs files is chosen at construction — App does not orchestrate two functions.Uploaderstill answers oneuploadmessage;Send + Synconly so several threads can deliver it.BatchPublisheris the sender that fans those messages out. Do not putupload_batchon the uploader. Do not make App a thread scheduler with closures over free functions.三个人的讨论
Kay: workers should
recordas each upload completes — live objects collaborating, closer to the old serial “history of what went up.”Ousterhout: that puts a mutex and two opens on the upload path. The issue asked for no extra disk bottleneck. Emitter already waits for the full ordered set; History is the same after-the-fact journal. On failure, some HTTP PUTs may already have happened — that is inherent to parallel I/O, not a reason to interleave logging.
Evans: the application “upload run” is the consistency boundary for output. Partial journals are not a domain invariant we promised.
--no-logcovering both files is the language fix; a second flag would be a worse split.Ousterhout: then BatchPublisher is acceptable only if Uploader stays one-artifact and concurrency is bounded. Reject rayon, reject 183 threads, reject skipping failed files and printing the rest.
Kay agrees: the extra object is the sender, not a second upload message. Concedes recording after the batch.
Evans: no Go 0.2 forward-compat; the old “history.log is still written” help text was the bug.
收敛结论
已收敛.
--no-log→History::silent()(neither file). OtherwiseHistory::fileswrites both, in input order, after a successful batch.BatchPublishersendsuploadwith 8 workers viathread::scope; 0–1 artifacts stay serial.Uploader: Send + Sync. First error fails the run; in-flight work finishes; further artifacts are not started. Output order preserved.Rejected:
--no-history— splits ubiquitous languageUploader::upload_batch— parallelism is the sender’s concernif no_logaround two free functions — shallow; two log paths leak决策与后续
Decided:
Residual risk: a failed batch may still have uploaded some in-flight files (same class of leftover as serial-stop-at-N). GitHub secondary rate limits under 8 concurrent Contents PUTs are possible; 8 is conservative versus 183 serial.