From 8b177e21a01580a49b2ee221a71eb56be68b2d7e Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Fri, 9 Oct 2026 18:28:48 -0400 Subject: [PATCH 1/9] Harden scanner accounting and expose controlled scan reports --- CHANGELOG.md | 19 ++- MANUAL.md | 21 ++- README.md | 14 +- sized.1 | 10 +- src/lib.rs | 227 +++++++------------------ src/main.rs | 4 +- src/scan.rs | 417 ++++++++++++++++++++++++++++++++++++++++++++++ tests/cli.rs | 52 ++++++ tests/scanning.rs | 254 ++++++++++++++++++++++++++++ 9 files changed, 839 insertions(+), 179 deletions(-) create mode 100644 src/scan.rs create mode 100644 tests/scanning.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ac0691..3d22d88 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Fixed +- Count hard-link allocation once in a deterministic traversal order while + retaining apparent sizes per pathname. Filtered comparison has independent + allocation ownership, including when the first link is ignored. +- Report filesystem errors and incomplete ancestor totals instead of silently + dropping errors or treating vanished entries as accessible empty directories. +- Inherit ignore rules when scanning nested directories; accept dangling + symlinks as scan targets and distinguish special files from directories. + +### Added +- A scanner module with structured reports, file identities, progress counters, + cooperative cancellation, and per-scan thread pools. +- Optional `-x` / `--one-file-system` boundary handling. Partial CLI reports + return a nonzero exit status and include `complete` in JSON output. + ## [1.0.0] - 2026-01-22 This stable release marks the official launch of `sized` under the GNU General Public License v3.0. @@ -19,4 +36,4 @@ This stable release marks the official launch of `sized` under the GNU General P - **Exporting**: Support for CSV and NDJSON output formats. - **Documentation**: Comprehensive standard Unix man pages and a detailed user guide. - **Shell Completions**: Automatic generation for Bash, Zsh, and Fish. -- **Licensing**: GPL-3.0 to ensure the tool remains a shared public resource. \ No newline at end of file +- **Licensing**: GPL-3.0 to ensure the tool remains a shared public resource. diff --git a/MANUAL.md b/MANUAL.md index 9c76905..8143987 100644 --- a/MANUAL.md +++ b/MANUAL.md @@ -35,9 +35,24 @@ sized [path] ``` ### Key Concepts -- **Disk Usage** (Default): The actual physical space consumed on disk (Blocks * 512 bytes). This is the "true" footprint and the metric `sized` prioritizes. +- **Disk Usage** (Default): Filesystem-reported allocation (Blocks * 512 bytes), with hard links counted once per scan. Shared extents and snapshots can make this differ from the space recovered by deletion. - **Apparent Size**: The logical size of the entry (file length). Available via the `-a` or `--apparent` flag. -- **Blocks**: The actual filesystem blocks allocated. +- **Blocks**: Filesystem-reported 512-byte units of allocation. Directory metadata is included. + +Symlinks are scanned as leaves, including dangling links; their targets are not +followed. Missing targets or unreadable entries are reported on stderr. Partial +scans return a nonzero exit status and JSON includes `complete: false`; inspect +that status before relying on a total. + +### Filesystem Boundary (`-x` / `--one-file-system`) + +Skip entries whose device differs from the scan root, useful when a directory +contains mounted filesystems. Boundary entries contribute no size, and JSON +marks them with `skipped_mount`. + +```bash +sized -x /path/to/volume +``` ### Path Display - **Relative Path** (Default): `sized` shows paths relative to the current directory. @@ -53,7 +68,7 @@ sized -d 1 ``` > [!NOTE] -> Regardless of the display depth, `sized` always calculates the *total* size of all subdirectories accurately by traversing the entire tree. +> Display depth limits output, not scanning. The tree is traversed unless filtered or excluded by a filesystem boundary; scan errors mark totals incomplete. ## Filtering and Sorting diff --git a/README.md b/README.md index 9a4e42e..cd0696d 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A fast, concurrent, and feature-rich command-line tool for visualizing disk usag ## Features -- **Physical by Default**: Prioritizes **Disk Usage** (actual blocks consumed) as the default metric, essential for accurately seeing how much storage is actually gone. +- **Allocation by Default**: Prioritizes filesystem-reported allocated blocks; sparse files retain their separate apparent size, and hard-link allocation is counted once per scan. Reported allocation is not a promise of bytes freed by deletion on filesystems with shared extents or snapshots. - **Fast & Concurrent**: Uses `rayon` to process directories in parallel, making it extremely fast on modern multi-core systems. - **Rich Output**: Beautifully formatted tables with colors, distinguishing files and directories. - **Apparent Size**: Toggle logical file length with `-a` or `--apparent`. @@ -98,6 +98,7 @@ sized /path/to/directory | `-f`, `--path-full` | Force absolute paths in headers | `sized -f` | | `-i`, `--ignore` | Respect .gitignore files | `sized -i` | | `-j`, `--threads` | Set number of threads | `sized -j 4` | +| `-x`, `--one-file-system` | Skip entries on other filesystems | `sized -x /` | | `--format` | Output format (text, csv, json) | `sized --format json` | | `--save` | Save output to file | `sized --save` | | `-c`, `--compare` | Compare total vs. non-ignored files | `sized -i -c` | @@ -137,6 +138,17 @@ autoload -Uz compinit && compinit sized --completions fish > ~/.config/fish/completions/sized.fish ``` +## Scan status and library API + +Scans report missing or unreadable entries on stderr and return a nonzero status +when incomplete. JSON reports include `complete` so automation can distinguish +a partial report from a fully scanned tree. Symlinks are not followed. + +Library users can call `scan_tree` with `ScanOptions` and a fresh `ScanControl` +for each request. Clones of the control expose progress and cancellation; +cancellation is cooperative between filesystem calls. `build_tree` remains as +a convenience helper, while `scan_tree` retains structured diagnostics. + ## License This project is licensed under the **GNU General Public License v3.0 (GPL-3.0)**. diff --git a/sized.1 b/sized.1 index a080f9d..77cfaf0 100644 --- a/sized.1 +++ b/sized.1 @@ -1,10 +1,10 @@ .ie \n(.g .ds Aq \(aq .el .ds Aq ' -.TH sized 1 "sized 1.0.0" +.TH sized 1 "sized 1.0.0" .SH NAME sized \- A modern, fast, and concurrent disk usage analyzer .SH SYNOPSIS -\fBsized\fR [\fB\-v\fR|\fB\-\-version\fR] [\fB\-m\fR|\fB\-\-min\-size\fR] [\fB\-n\fR|\fB\-\-number\fR] [\fB\-\-sort\fR] [\fB\-a\fR|\fB\-\-apparent\fR] [\fB\-d\fR|\fB\-\-depth\fR] [\fB\-f\fR|\fB\-\-path\-full\fR] [\fB\-\-path\-relative\fR] [\fB\-j\fR|\fB\-\-threads\fR] [\fB\-\-completions\fR] [\fB\-i\fR|\fB\-\-ignore\fR] [\fB\-\-format\fR] [\fB\-\-save\fR] [\fB\-\-precision\fR] [\fB\-\-units\fR] [\fB\-c\fR|\fB\-\-compare\fR] [\fB\-h\fR|\fB\-\-help\fR] [\fIPATH\fR] +\fBsized\fR [\fB\-v\fR|\fB\-\-version\fR] [\fB\-m\fR|\fB\-\-min\-size\fR] [\fB\-n\fR|\fB\-\-number\fR] [\fB\-\-sort\fR] [\fB\-a\fR|\fB\-\-apparent\fR] [\fB\-d\fR|\fB\-\-depth\fR] [\fB\-f\fR|\fB\-\-path\-full\fR] [\fB\-\-path\-relative\fR] [\fB\-j\fR|\fB\-\-threads\fR] [\fB\-x\fR|\fB\-\-one\-file\-system\fR] [\fB\-\-completions\fR] [\fB\-i\fR|\fB\-\-ignore\fR] [\fB\-\-format\fR] [\fB\-\-save\fR] [\fB\-\-precision\fR] [\fB\-\-units\fR] [\fB\-c\fR|\fB\-\-compare\fR] [\fB\-h\fR|\fB\-\-help\fR] [\fIPATH\fR] .SH DESCRIPTION A modern, fast, and concurrent disk usage analyzer .SH OPTIONS @@ -36,6 +36,9 @@ Display paths relative to current directory (default) \fB\-j\fR, \fB\-\-threads\fR \fI\fR Number of threads to use (defaults to available logical CPUs) .TP +\fB\-x\fR, \fB\-\-one\-file\-system\fR +Stay on the target\*(Aqs filesystem instead of descending into other mounts +.TP \fB\-\-completions\fR \fI\fR Generate shell completions .br @@ -76,6 +79,3 @@ Print help Directory to analyze .SH VERSION v1.0.0 - -.SH COPYRIGHT -sized is licensed under the GNU General Public License v3.0 (GPL-3.0). diff --git a/src/lib.rs b/src/lib.rs index 37ce7d0..3925ba8 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -6,15 +6,17 @@ use colored::*; use comfy_table::presets::UTF8_FULL; use comfy_table::{Attribute, Cell, CellAlignment, Color, ContentArrangement, Table}; use csv::WriterBuilder; -use ignore::WalkBuilder; -use rayon::prelude::*; -use serde::Serialize; use std::cmp::Ordering; use std::io::Write; -use std::os::unix::fs::MetadataExt; use std::path::{Path, PathBuf}; use std::str::FromStr; +pub mod scan; +pub use scan::{ + scan_tree, EntryType, FileIdentity, Node, ScanControl, ScanError, ScanIssue, ScanOptions, + ScanReport, +}; + #[derive(Parser, Debug, Clone)] #[command(author, version, about, long_about = None)] #[command(disable_version_flag = true)] @@ -61,6 +63,10 @@ pub struct Args { #[arg(short = 'j', long = "threads")] pub threads: Option, + /// Stay on the target's filesystem instead of descending into other mounts + #[arg(short = 'x', long)] + pub one_file_system: bool, + /// Generate shell completions #[arg(long, value_enum)] pub completions: Option, @@ -147,40 +153,14 @@ impl FromStr for SortDirection { } } -#[derive(Clone, Serialize, Debug)] -pub struct Node { - pub path: PathBuf, - pub size_bytes: u64, - pub blocks: u64, - pub size_bytes_filtered: u64, - pub blocks_filtered: u64, - pub entry_type: EntryType, - pub accessible: bool, - #[serde(skip)] - pub children: Vec, -} - -#[derive(Clone, Serialize, Debug, PartialEq, Eq, PartialOrd, Ord)] -pub enum EntryType { - File, - Dir, -} - -impl std::fmt::Display for EntryType { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - EntryType::File => write!(f, "File"), - EntryType::Dir => write!(f, "Dir"), - } - } -} - -pub fn run(args: Args, mut writer: &mut dyn Write) { +/// Returns whether the scan completed without missing entries. The caller +/// chooses the exit status; partial reports remain available to the user. +pub fn run(args: Args, mut writer: &mut dyn Write) -> bool { if let Some(shell) = args.completions { let mut cmd = Args::command(); let name = cmd.get_name().to_string(); generate(shell, &mut cmd, name, &mut std::io::stdout()); - return; + return true; } if let Some(out_dir) = args.generate_man_page { @@ -193,23 +173,11 @@ pub fn run(args: Args, mut writer: &mut dyn Write) { let file_path = out_dir.join("sized.1"); std::fs::write(&file_path, buffer).expect("Failed to write man page"); println!("Man page generated at {}", file_path.display()); - return; - } - - if let Some(threads) = args.threads { - rayon::ThreadPoolBuilder::new() - .num_threads(threads) - .build_global() - .ok(); // Ignore error if initialized called multiple times (e.g. in tests) + return true; } let target_path = args.path.clone(); - if !target_path.exists() { - eprintln!("Error: Path '{}' does not exist.", target_path.display()); - std::process::exit(1); - } - let min_bytes = if let Some(size_str) = &args.min_size { match Byte::parse_str(size_str, true) { Ok(byte) => byte.as_u64(), @@ -234,134 +202,55 @@ pub fn run(args: Args, mut writer: &mut dyn Write) { } } - let root_node = build_tree(&target_path, args.ignore, args.compare); + let report = match scan_tree( + &target_path, + &ScanOptions { + respect_ignore: args.ignore, + compare_ignore: args.compare, + stay_on_filesystem: args.one_file_system, + threads: args.threads, + }, + &ScanControl::default(), + ) { + Ok(report) => report, + Err(e) => { + eprintln!("Error: {e}"); + return false; + } + }; + for issue in &report.issues { + eprintln!( + "Warning: incomplete scan at '{}': {}", + issue.path.display(), + issue.message + ); + } + let root_node = report.root; if root_node.size_bytes < min_bytes { if args.format == OutputFormat::Text { writeln!(writer, "Root directory is smaller than minimum size.").ok(); } - return; + return root_node.complete; } process_node_recursive(&mut writer, &root_node, 0, &args, min_bytes, &sort_criteria); + root_node.complete } pub fn build_tree(path: &Path, ignore: bool, compare: bool) -> Node { - let metadata = path.symlink_metadata(); - - if let Ok(meta) = metadata { - // Treat symlinks as files (nodes) but do not recurse - if meta.is_file() || meta.is_symlink() { - return Node { - path: path.to_path_buf(), - size_bytes: meta.len(), - blocks: meta.blocks(), - size_bytes_filtered: meta.len(), // Single file is its own filtered size for now - blocks_filtered: meta.blocks(), - entry_type: EntryType::File, - accessible: true, - children: vec![], - }; - } - } - - // Check if we can read the directory (handle permissions) - if let Err(e) = std::fs::read_dir(path) { - if e.kind() == std::io::ErrorKind::PermissionDenied { - // Return an "empty" directory node marked as inaccessible - let meta = path.symlink_metadata().ok(); - let size = meta.as_ref().map(|m| m.len()).unwrap_or(0); - let blocks = meta.as_ref().map(|m| m.blocks()).unwrap_or(0); - return Node { - path: path.to_path_buf(), - size_bytes: size, - blocks, - size_bytes_filtered: size, - blocks_filtered: blocks, - entry_type: EntryType::Dir, - accessible: false, - children: vec![], - }; - } - } - - // Total walker (always everything if compare is true, else respects 'ignore' arg) - let total_ignore = if compare { false } else { ignore }; - let walker_total = WalkBuilder::new(path) - .standard_filters(false) - .hidden(false) - .git_ignore(total_ignore) - .ignore(total_ignore) - .max_depth(Some(1)) - .build(); - - let child_paths_total: Vec = walker_total - .into_iter() - .filter_map(|e| e.ok()) - .filter(|e| e.path() != path) - .map(|e| e.path().to_path_buf()) - .collect(); - - // Filtered set (only if compare is true) - let non_ignored_set: std::collections::HashSet = if compare { - let walker_filtered = WalkBuilder::new(path) - .standard_filters(false) - .hidden(false) - .git_ignore(true) - .ignore(true) - .max_depth(Some(1)) - .build(); - - walker_filtered - .into_iter() - .filter_map(|e| e.ok()) - .filter(|e| e.path() != path) - .map(|e| e.path().to_path_buf()) - .collect() - } else { - std::collections::HashSet::new() - }; - - let children: Vec = child_paths_total - .par_iter() - .map(|p| build_tree(p, ignore, compare)) - .collect(); - - let mut size_bytes = 0; - let mut blocks = 0; - let mut size_bytes_filtered = 0; - let mut blocks_filtered = 0; - - for child in &children { - size_bytes += child.size_bytes; - blocks += child.blocks; - - if compare { - if non_ignored_set.contains(&child.path) { - size_bytes_filtered += child.size_bytes_filtered; - blocks_filtered += child.blocks_filtered; - } - } else { - size_bytes_filtered += child.size_bytes_filtered; - blocks_filtered += child.blocks_filtered; - } - } - - let (self_size, self_blocks) = path - .symlink_metadata() - .map(|m| (m.len(), m.blocks())) - .unwrap_or((0, 0)); - - Node { - path: path.to_path_buf(), - size_bytes: size_bytes + self_size, - blocks: blocks + self_blocks, - size_bytes_filtered: size_bytes_filtered + self_size, // self is always part of self - blocks_filtered: blocks_filtered + self_blocks, - entry_type: EntryType::Dir, - accessible: true, - children, - } + // Compatibility helper; callers needing diagnostics should use scan_tree. + scan_tree( + path, + &ScanOptions { + respect_ignore: ignore, + compare_ignore: compare, + ..ScanOptions::default() + }, + &ScanControl::default(), + ) + .map(|report| report.root) + .unwrap_or_else(|_| Node::unavailable(path)) } fn process_node_recursive( @@ -372,8 +261,6 @@ fn process_node_recursive( min_bytes: u64, sort_criteria: &[(SortColumn, SortDirection)], ) { - if node.children.is_empty() && node.entry_type == EntryType::Dir {} - let mut display_children: Vec<&Node> = node .children .iter() @@ -406,7 +293,7 @@ fn process_node_recursive( print_output( writer, - &node, + node, &display_children, args, &relative_path, @@ -735,7 +622,7 @@ fn print_text( Cell::new("N/A") .fg(Color::Red) .set_alignment(CellAlignment::Right), - Cell::new("Access Denied") + Cell::new("Unavailable") .fg(Color::Red) .set_alignment(CellAlignment::Right), ]; @@ -794,6 +681,9 @@ fn print_json(writer: &mut dyn Write, parent_node: &Node, children: &[&Node], ar "path": &c.path, "entry_type": c.entry_type.to_string(), "accessible": c.accessible, + "complete": c.complete, + "hard_link_duplicate": c.hard_link_duplicate, + "skipped_mount": c.skipped_mount, "disk_usage": c.blocks * 512, "blocks": c.blocks, }); @@ -831,6 +721,7 @@ fn print_json(writer: &mut dyn Write, parent_node: &Node, children: &[&Node], ar "total_blocks": parent_node.blocks, "entries": entries_view, "accessible": parent_node.accessible, + "complete": parent_node.complete, }); if args.apparent { diff --git a/src/main.rs b/src/main.rs index 4bd87fd..a3a2edc 100644 --- a/src/main.rs +++ b/src/main.rs @@ -29,5 +29,7 @@ fn main() { Box::new(std::io::stdout()) }; - run(args, &mut writer); + if !run(args, &mut writer) { + std::process::exit(1); + } } diff --git a/src/scan.rs b/src/scan.rs new file mode 100644 index 0000000..8dd2ec1 --- /dev/null +++ b/src/scan.rs @@ -0,0 +1,417 @@ +//! Filesystem scanning, independent of CLI parsing and presentation. +//! Allocation is the filesystem's reported 512-byte blocks, not a prediction +//! of bytes freed by deletion (snapshots and shared extents can differ). +use ignore::WalkBuilder; +use rayon::prelude::*; +use serde::Serialize; +use std::{ + collections::HashSet, + fs::{self, Metadata}, + io, + os::unix::fs::MetadataExt, + path::{Path, PathBuf}, + sync::{ + atomic::{AtomicBool, AtomicU64, Ordering}, + Arc, Mutex, + }, +}; + +#[derive(Clone, Copy, Serialize, Debug, PartialEq, Eq)] +pub struct FileIdentity { + pub device: u64, + pub inode: u64, + pub links: u64, +} + +#[derive(Clone, Serialize, Debug)] +pub struct Node { + pub path: PathBuf, + pub size_bytes: u64, + pub blocks: u64, + pub size_bytes_filtered: u64, + pub blocks_filtered: u64, + pub entry_type: EntryType, + pub accessible: bool, + /// False when this node or any descendant could not be scanned. + pub complete: bool, + pub symlink: bool, + pub skipped_mount: bool, + pub hard_link_duplicate: bool, + pub identity: Option, + #[serde(skip)] + pub children: Vec, + #[serde(skip)] + pub own_size_bytes: u64, + #[serde(skip)] + pub own_blocks: u64, + #[serde(skip)] + included_by_filter: bool, +} + +#[derive(Clone, Serialize, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum EntryType { + File, + Dir, + Special, + Unknown, +} + +impl std::fmt::Display for EntryType { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{:?}", self) + } +} + +impl Node { + pub(crate) fn unavailable(path: &Path) -> Self { + Self { + path: path.into(), + size_bytes: 0, + blocks: 0, + size_bytes_filtered: 0, + blocks_filtered: 0, + entry_type: EntryType::Unknown, + accessible: false, + complete: false, + symlink: false, + skipped_mount: false, + hard_link_duplicate: false, + identity: None, + children: Vec::new(), + own_size_bytes: 0, + own_blocks: 0, + included_by_filter: true, + } + } + + fn from_metadata(path: &Path, meta: &Metadata) -> Self { + let mut node = Self::unavailable(path); + node.entry_type = if meta.is_dir() { + EntryType::Dir + } else if meta.is_file() || meta.is_symlink() { + EntryType::File + } else { + EntryType::Special + }; + node.accessible = true; + node.complete = true; + node.symlink = meta.is_symlink(); + node.identity = Some(FileIdentity { + device: meta.dev(), + inode: meta.ino(), + links: meta.nlink(), + }); + node.own_size_bytes = meta.len(); + node.own_blocks = meta.blocks(); + node + } +} + +#[derive(Clone, Debug, Default)] +pub struct ScanOptions { + pub respect_ignore: bool, + pub compare_ignore: bool, + pub stay_on_filesystem: bool, + /// None or zero selects Rayon's default worker count, scoped to this scan. + pub threads: Option, +} + +/// Create a fresh control for each scan; clones share cancellation and progress. +#[derive(Clone, Debug, Default)] +pub struct ScanControl { + cancelled: Arc, + visited: Arc, +} + +impl ScanControl { + pub fn cancel(&self) { + self.cancelled.store(true, Ordering::Relaxed); + } + pub fn is_cancelled(&self) -> bool { + self.cancelled.load(Ordering::Relaxed) + } + pub fn entries_scanned(&self) -> u64 { + self.visited.load(Ordering::Relaxed) + } + fn check(&self) -> Result<(), ScanError> { + if self.is_cancelled() { + Err(ScanError::Cancelled) + } else { + Ok(()) + } + } +} + +#[derive(Debug)] +pub struct ScanIssue { + pub path: PathBuf, + pub kind: io::ErrorKind, + pub message: String, +} + +#[derive(Debug)] +pub struct ScanReport { + pub root: Node, + pub issues: Vec, + pub entries_scanned: u64, +} + +#[derive(Debug)] +pub enum ScanError { + Root { path: PathBuf, source: io::Error }, + WorkerPool(rayon::ThreadPoolBuildError), + Cancelled, +} + +impl std::fmt::Display for ScanError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Root { path, source } => { + write!(f, "cannot scan '{}': {}", path.display(), source) + } + Self::WorkerPool(e) => write!(f, "cannot start scan workers: {e}"), + Self::Cancelled => write!(f, "scan cancelled"), + } + } +} +impl std::error::Error for ScanError {} + +/// The completed tree is published only after deterministic hard-link accounting. +/// Cancellation is cooperative between filesystem calls, not syscall preemption. +pub fn scan_tree( + path: &Path, + options: &ScanOptions, + control: &ScanControl, +) -> Result { + control.check()?; + let metadata = fs::symlink_metadata(path).map_err(|source| ScanError::Root { + path: path.into(), + source, + })?; + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(options.threads.unwrap_or(0)) + .build() + .map_err(ScanError::WorkerPool)?; + let context = Context { + options, + control, + root_device: metadata.dev(), + issues: Mutex::new(Vec::new()), + }; + let mut root = pool.install(|| context.visit(path, Some(metadata)))?; + control.check()?; + reduce( + &mut root, + true, + &mut HashSet::new(), + &mut HashSet::new(), + control, + )?; + control.check()?; + let mut issues = context.issues.into_inner().unwrap(); + issues.sort_by(|a, b| a.path.cmp(&b.path).then(a.message.cmp(&b.message))); + Ok(ScanReport { + root, + issues, + entries_scanned: control.entries_scanned(), + }) +} + +struct Context<'a> { + options: &'a ScanOptions, + control: &'a ScanControl, + root_device: u64, + issues: Mutex>, +} + +impl Context<'_> { + fn issue(&self, path: &Path, kind: io::ErrorKind, message: String) { + self.issues.lock().unwrap().push(ScanIssue { + path: path.into(), + kind, + message, + }); + } + + fn visit(&self, path: &Path, metadata: Option) -> Result { + self.control.check()?; + self.control.visited.fetch_add(1, Ordering::Relaxed); + let metadata = match metadata + .map(Ok) + .unwrap_or_else(|| fs::symlink_metadata(path)) + { + Ok(meta) => meta, + Err(e) => { + self.issue(path, e.kind(), e.to_string()); + return Ok(Node::unavailable(path)); + } + }; + let mut node = Node::from_metadata(path, &metadata); + if self.options.stay_on_filesystem && metadata.dev() != self.root_device { + node.skipped_mount = true; + node.own_size_bytes = 0; + node.own_blocks = 0; + return Ok(node); + } + if node.entry_type != EntryType::Dir { + return Ok(node); + } + if let Err(e) = fs::read_dir(path) { + self.issue(path, e.kind(), e.to_string()); + node.accessible = false; + node.complete = false; + return Ok(node); + } + let total_ignore = self.options.respect_ignore && !self.options.compare_ignore; + let (paths, total_complete) = self.paths(path, total_ignore)?; + let (filtered, filtered_complete) = if self.options.compare_ignore { + let (paths, complete) = self.paths(path, true)?; + (paths.into_iter().collect::>(), complete) + } else { + (HashSet::new(), true) + }; + node.complete = total_complete && filtered_complete; + node.children = paths + .par_iter() + .map(|child| { + let mut node = self.visit(child, None)?; + node.included_by_filter = !self.options.compare_ignore || filtered.contains(child); + Ok(node) + }) + .collect::, ScanError>>()?; + Ok(node) + } + + fn paths(&self, path: &Path, respect_ignore: bool) -> Result<(Vec, bool), ScanError> { + let walker = WalkBuilder::new(path) + .standard_filters(false) + .hidden(false) + .parents(respect_ignore) + .git_ignore(respect_ignore) + .ignore(respect_ignore) + .max_depth(Some(1)) + .build(); + let mut paths = Vec::new(); + let mut complete = true; + for entry in walker { + self.control.check()?; + match entry { + Ok(entry) if entry.path() != path => paths.push(entry.into_path()), + Ok(_) => {} + Err(e) => { + complete = false; + self.issue( + path, + e.io_error().map_or(io::ErrorKind::Other, |e| e.kind()), + e.to_string(), + ); + } + } + } + // Parallel collection preserves this order; ownership of a shared inode + // is then stable across worker counts and repeated scans. + paths.sort(); + Ok((paths, complete)) + } +} + +fn reduce( + node: &mut Node, + included: bool, + all_seen: &mut HashSet<(u64, u64)>, + filtered_seen: &mut HashSet<(u64, u64)>, + control: &ScanControl, +) -> Result<(), ScanError> { + control.check()?; + node.size_bytes = node.own_size_bytes; + node.blocks = node.own_blocks; + node.size_bytes_filtered = if included { node.own_size_bytes } else { 0 }; + node.blocks_filtered = if included { node.own_blocks } else { 0 }; + if node.entry_type != EntryType::Dir { + if let Some(id) = node.identity.filter(|id| id.links > 1) { + let key = (id.device, id.inode); + if !all_seen.insert(key) { + node.blocks = 0; + node.hard_link_duplicate = true; + } + if included && !filtered_seen.insert(key) { + node.blocks_filtered = 0; + } + } + } + for child in &mut node.children { + reduce( + child, + included && child.included_by_filter, + all_seen, + filtered_seen, + control, + )?; + node.size_bytes += child.size_bytes; + node.blocks += child.blocks; + node.size_bytes_filtered += child.size_bytes_filtered; + node.blocks_filtered += child.blocks_filtered; + node.complete &= child.complete; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::tempdir; + + #[test] + fn foreign_device_entry_is_marked_without_scanning_its_contents() { + let dir = tempdir().unwrap(); + fs::write(dir.path().join("content"), b"not visited").unwrap(); + let meta = fs::symlink_metadata(dir.path()).unwrap(); + let control = ScanControl::default(); + let options = ScanOptions { + stay_on_filesystem: true, + ..ScanOptions::default() + }; + // Inject the parent scan's different device; this exercises the boundary + // policy without creating mounts or requiring privileged test setup. + let context = Context { + options: &options, + control: &control, + root_device: meta.dev().wrapping_add(1), + issues: Mutex::new(Vec::new()), + }; + let mut node = context.visit(dir.path(), Some(meta)).unwrap(); + reduce( + &mut node, + true, + &mut HashSet::new(), + &mut HashSet::new(), + &control, + ) + .unwrap(); + assert!(node.skipped_mount); + assert!(node.children.is_empty()); + assert_eq!(node.blocks, 0); + assert_eq!(control.entries_scanned(), 1); + assert!(node.complete); + } + + #[test] + fn vanished_child_is_reported_without_fabricating_accessibility() { + let dir = tempdir().unwrap(); + let path = dir.path().join("vanished-after-discovery"); + let options = ScanOptions::default(); + let control = ScanControl::default(); + let context = Context { + options: &options, + control: &control, + root_device: fs::metadata(dir.path()).unwrap().dev(), + issues: Mutex::new(Vec::new()), + }; + let node = context.visit(&path, None).unwrap(); + assert!(!node.accessible); + assert!(!node.complete); + let issues = context.issues.into_inner().unwrap(); + assert_eq!(issues[0].path, path); + assert_eq!(issues[0].kind, io::ErrorKind::NotFound); + } +} diff --git a/tests/cli.rs b/tests/cli.rs index 11d9c4c..a2b6381 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -45,3 +45,55 @@ fn test_json_output() { .stdout(predicate::str::contains("\"entry_type\":\"File\"")) .stdout(predicate::str::contains("\"path\":")); } + +#[test] +fn missing_path_fails_without_a_success_report() { + let dir = TempDir::new().unwrap(); + Command::new(env!("CARGO_BIN_EXE_sized")) + .arg(dir.path().join("missing")) + .args(["--format", "json"]) + .assert() + .failure() + .stdout(predicate::str::is_empty()) + .stderr(predicate::str::contains("cannot scan")); +} + +#[cfg(unix)] +#[test] +fn partial_scan_has_nonzero_status_and_explicit_json_flag() { + use std::{fs, os::unix::fs::PermissionsExt}; + let dir = TempDir::new().unwrap(); + let blocked = dir.path().join("blocked"); + fs::create_dir(&blocked).unwrap(); + fs::set_permissions(&blocked, fs::Permissions::from_mode(0o000)).unwrap(); + let result = std::panic::catch_unwind(|| { + assert!( + fs::read_dir(&blocked).is_err(), + "permission tests require an unprivileged user" + ); + Command::new(env!("CARGO_BIN_EXE_sized")) + .arg(dir.path()) + .args(["--format", "json"]) + .assert() + .failure() + .stdout(predicate::str::contains("\"complete\":false")) + .stderr(predicate::str::contains("incomplete scan")); + }); + fs::set_permissions(blocked, fs::Permissions::from_mode(0o700)).unwrap(); + result.unwrap(); +} + +#[cfg(unix)] +#[test] +fn dangling_symlink_is_a_valid_scan_target() { + use std::os::unix::fs::symlink; + let dir = TempDir::new().unwrap(); + let path = dir.path().join("dangling"); + symlink(dir.path().join("missing"), &path).unwrap(); + Command::new(env!("CARGO_BIN_EXE_sized")) + .arg(path) + .args(["--format", "json"]) + .assert() + .success() + .stdout(predicate::str::contains("\"complete\":true")); +} diff --git a/tests/scanning.rs b/tests/scanning.rs new file mode 100644 index 0000000..5b62750 --- /dev/null +++ b/tests/scanning.rs @@ -0,0 +1,254 @@ +#![cfg(unix)] +use sized::{build_tree, scan_tree, EntryType, ScanControl, ScanError, ScanOptions}; +use std::{ + fs, + io::Write, + os::unix::fs::{symlink, MetadataExt, PermissionsExt}, +}; +use tempfile::tempdir; + +fn scan(path: &std::path::Path) -> sized::ScanReport { + scan_tree(path, &ScanOptions::default(), &ScanControl::default()).unwrap() +} + +#[test] +fn hard_links_count_allocation_once_but_keep_apparent_sizes() { + let dir = tempdir().unwrap(); + let original = dir.path().join("a-original"); + fs::write(&original, [9; 65536]).unwrap(); + fs::hard_link(&original, dir.path().join("z-alias")).unwrap(); + let expected = + fs::metadata(&original).unwrap().blocks() + fs::metadata(dir.path()).unwrap().blocks(); + for threads in [1, 4] { + let report = scan_tree( + dir.path(), + &ScanOptions { + threads: Some(threads), + ..ScanOptions::default() + }, + &ScanControl::default(), + ) + .unwrap(); + assert_eq!(report.root.blocks, expected); + assert_eq!(report.root.children[0].size_bytes, 65536); + assert_eq!(report.root.children[1].size_bytes, 65536); + assert!(!report.root.children[0].hard_link_duplicate); + assert!(report.root.children[1].hard_link_duplicate); + assert_eq!(report.root.children[1].blocks, 0); + assert_eq!(report.entries_scanned, 3); + assert!(report.root.complete); + } +} + +#[test] +fn ignored_hard_link_does_not_steal_filtered_allocation() { + let dir = tempdir().unwrap(); + fs::write(dir.path().join(".ignore"), "a-ignored\n").unwrap(); + let original = dir.path().join("a-ignored"); + fs::write(&original, [9; 65536]).unwrap(); + fs::hard_link(&original, dir.path().join("z-included")).unwrap(); + let report = scan_tree( + dir.path(), + &ScanOptions { + compare_ignore: true, + ..ScanOptions::default() + }, + &ScanControl::default(), + ) + .unwrap(); + let expected = fs::metadata(&original).unwrap().blocks() + + fs::metadata(dir.path()).unwrap().blocks() + + fs::metadata(dir.path().join(".ignore")).unwrap().blocks(); + assert_eq!(report.root.blocks, expected); + assert_eq!(report.root.blocks_filtered, expected); + assert_eq!(report.root.children.last().unwrap().blocks, 0); + assert!(report.root.children.last().unwrap().blocks_filtered > 0); +} + +#[test] +fn hidden_files_are_counted_without_ignore_filtering() { + let dir = tempdir().unwrap(); + fs::write(dir.path().join(".hidden"), b"hidden").unwrap(); + fs::write(dir.path().join(".ignore"), "build\n").unwrap(); + fs::write(dir.path().join("build"), b"build").unwrap(); + assert_eq!(scan(dir.path()).root.children.len(), 3); + let report = scan_tree( + dir.path(), + &ScanOptions { + respect_ignore: true, + ..ScanOptions::default() + }, + &ScanControl::default(), + ) + .unwrap(); + assert_eq!(report.root.children.len(), 2); + assert!(report + .root + .children + .iter() + .any(|n| n.path.ends_with(".hidden"))); +} + +#[test] +fn ignore_rules_are_inherited_by_nested_directories() { + let dir = tempdir().unwrap(); + fs::write(dir.path().join(".ignore"), "*.tmp\n").unwrap(); + let nested = dir.path().join("nested"); + fs::create_dir(&nested).unwrap(); + fs::write(nested.join("skip.tmp"), b"ignored").unwrap(); + fs::write(nested.join("keep.bin"), b"included").unwrap(); + let report = scan_tree( + dir.path(), + &ScanOptions { + respect_ignore: true, + ..ScanOptions::default() + }, + &ScanControl::default(), + ) + .unwrap(); + let children = &report + .root + .children + .iter() + .find(|n| n.path == nested) + .unwrap() + .children; + assert_eq!(children.len(), 1); + assert!(children[0].path.ends_with("keep.bin")); + let compared = scan_tree( + dir.path(), + &ScanOptions { + compare_ignore: true, + ..ScanOptions::default() + }, + &ScanControl::default(), + ) + .unwrap(); + assert_eq!(compared.root.size_bytes_filtered, report.root.size_bytes); + assert_eq!(compared.root.blocks_filtered, report.root.blocks); +} + +#[test] +fn sparse_file_reports_blocks_and_logical_length_separately() { + let dir = tempdir().unwrap(); + let path = dir.path().join("sparse"); + let mut file = fs::File::create(&path).unwrap(); + file.write_all(&[7; 4096]).unwrap(); + file.set_len(128 * 1024 * 1024).unwrap(); + file.sync_all().unwrap(); + let node = scan(&path).root; + assert_eq!(node.size_bytes, 128 * 1024 * 1024); + assert_eq!(node.blocks, fs::metadata(path).unwrap().blocks()); + assert!(node.blocks * 512 < node.size_bytes); +} + +#[test] +fn symlinks_including_loop_and_dangling_link_are_leaves() { + let dir = tempdir().unwrap(); + symlink(dir.path(), dir.path().join("loop")).unwrap(); + symlink(dir.path().join("missing"), dir.path().join("dangling")).unwrap(); + let report = scan(dir.path()); + assert_eq!(report.root.children.len(), 2); + for node in report.root.children { + assert!(node.symlink); + assert!(node.children.is_empty()); + assert!(node.complete); + } +} + +#[test] +fn missing_root_returns_error_and_legacy_helper_marks_unavailable() { + let dir = tempdir().unwrap(); + let path = dir.path().join("vanished"); + assert!( + matches!(scan_tree(&path, &ScanOptions::default(), &ScanControl::default()), + Err(ScanError::Root { source, .. }) if source.kind() == std::io::ErrorKind::NotFound) + ); + let node = build_tree(&path, false, false); + assert_eq!(node.entry_type, EntryType::Unknown); + assert!(!node.accessible); + assert!(!node.complete); +} + +#[test] +fn permission_error_propagates_incomplete_status_to_root() { + let dir = tempdir().unwrap(); + let blocked = dir.path().join("blocked"); + fs::create_dir(&blocked).unwrap(); + fs::write(blocked.join("unreadable"), b"data").unwrap(); + fs::set_permissions(&blocked, fs::Permissions::from_mode(0o000)).unwrap(); + let result = std::panic::catch_unwind(|| { + assert!( + fs::read_dir(&blocked).is_err(), + "permission tests require an unprivileged user" + ); + let report = scan(dir.path()); + assert!(report.root.accessible); + assert!(!report.root.complete); + assert!(!report.root.children[0].accessible); + assert_eq!(report.issues.len(), 1); + assert_eq!(report.issues[0].path, blocked); + assert_eq!(report.issues[0].kind, std::io::ErrorKind::PermissionDenied); + }); + fs::set_permissions(blocked, fs::Permissions::from_mode(0o700)).unwrap(); + result.unwrap(); +} + +#[test] +fn cancelled_scan_never_returns_completed_tree() { + let dir = tempdir().unwrap(); + let control = ScanControl::default(); + control.cancel(); + assert!(matches!( + scan_tree(dir.path(), &ScanOptions::default(), &control), + Err(ScanError::Cancelled) + )); + assert_eq!(control.entries_scanned(), 0); +} + +#[test] +fn cancellation_and_progress_work_during_scan() { + let dir = tempdir().unwrap(); + for i in 0..2000 { + fs::write(dir.path().join(format!("file-{i}")), b"x").unwrap(); + } + let control = ScanControl::default(); + let watcher = control.clone(); + let cancel = std::thread::spawn(move || { + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while watcher.entries_scanned() < 10 { + assert!( + std::time::Instant::now() < deadline, + "scan made no progress" + ); + std::thread::yield_now(); + } + watcher.cancel(); + }); + let result = scan_tree( + dir.path(), + &ScanOptions { + threads: Some(1), + ..ScanOptions::default() + }, + &control, + ); + cancel.join().unwrap(); + assert!(matches!(result, Err(ScanError::Cancelled))); + assert!(control.entries_scanned() >= 10); +} + +#[test] +fn special_file_is_not_treated_as_directory() { + let dir = tempdir().unwrap(); + let path = dir.path().join("fifo"); + assert!(std::process::Command::new("mkfifo") + .arg(&path) + .status() + .unwrap() + .success()); + let node = scan(&path).root; + assert_eq!(node.entry_type, EntryType::Special); + assert!(node.children.is_empty()); + assert!(node.complete); +} -- 2.54.0 From 988aad99510c0424cf0db5a116d9086183da74a5 Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Fri, 9 Oct 2026 18:32:37 -0400 Subject: [PATCH 2/9] Add reproducible unprivileged Linux build and mount checks --- CHANGELOG.md | 2 + README.md | 23 ++++++++ SHIPMENT.md | 25 +++++++++ benches/benchmark.rs | 6 +- scripts/check-linux-container.sh | 26 +++++++++ scripts/check-linux.sh | 27 +++++++++ scripts/linux-check.Dockerfile | 3 + src/lib.rs | 10 ++-- src/main.rs | 2 +- src/scan.rs | 2 +- tests/linux_mount.rs | 96 ++++++++++++++++++++++++++++++++ 11 files changed, 212 insertions(+), 10 deletions(-) create mode 100755 scripts/check-linux-container.sh create mode 100755 scripts/check-linux.sh create mode 100644 scripts/linux-check.Dockerfile create mode 100644 tests/linux_mount.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d22d88..74df2ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 cooperative cancellation, and per-scan thread pools. - Optional `-x` / `--one-file-system` boundary handling. Partial CLI reports return a nonzero exit status and include `complete` in JSON output. +- Repeatable Linux ARM64/AMD64 Docker checks with unprivileged permission + tests, a real mounted-filesystem boundary and an optimized-binary smoke test. ## [1.0.0] - 2026-01-22 diff --git a/README.md b/README.md index cd0696d..cedfbfc 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,29 @@ for each request. Clones of the control expose progress and cancellation; cancellation is cooperative between filesystem calls. `build_tree` remains as a convenience helper, while `scan_tree` retains structured diagnostics. +## Development checks + +Run `cargo test --locked` locally as an unprivileged user. For Linux build, +permission, filesystem-boundary and release smoke checks with Docker: + +```bash +./scripts/check-linux.sh linux/arm64 +./scripts/check-linux.sh linux/amd64 +``` + +The script pins the official Rust 1.88.0 Bookworm image by digest, adds rustfmt +and Clippy, then runs as UID 65532 with dropped capabilities. Source is mounted +read-only and copied into the temporary container. Fixtures live on Linux +filesystems, including two distinct tmpfs mounts; no privileged container or +host directory scan is required. AMD64 on an ARM64 host requires emulation. +Logs are saved in `target/linux-checks/`; containers and their build output are +removed on exit. Docker retains the reusable check images/build cache. +`SIZED_LINUX_JOBS` changes the default two build workers; `SIZED_LINUX_IMAGE` +can select a different toolchain image for an explicit compatibility check. +These are backend checks; desktop X11/Wayland acceptance belongs to SizeQueen. + +See [source review and release process](SHIPMENT.md) and the [live queue](TODO.md). + ## License This project is licensed under the **GNU General Public License v3.0 (GPL-3.0)**. diff --git a/SHIPMENT.md b/SHIPMENT.md index c5ed164..fdd2646 100644 --- a/SHIPMENT.md +++ b/SHIPMENT.md @@ -2,6 +2,31 @@ This document defines the process for versioning and distributing the `sized` project independently of the Git hosting provider (GitHub, GitLab, Gitea). +## Source review before a release + +Use a named branch from `main` and keep a pull request focused on one outcome. +The `sizequeen-scan-hardening` branch corrects accounting/error handling and +adds the scan controls needed by SizeQueen; shared-crate extraction follows +in a separate PR so reviewers can distinguish behaviour changes from packaging. + +1. Run local locked tests and strict Clippy as an unprivileged user. Run + `./scripts/check-linux.sh linux/arm64` and + `./scripts/check-linux.sh linux/amd64` for the repeatable Linux checks, + including actual mount boundaries and an optimized-binary smoke test. +2. Push the review branch. Open a PR against `main` when its scope is ready, + explaining changed behaviour, test evidence and remaining limits. For this + branch, call out nonzero status on partial scans, added JSON status fields, + and library API changes. Record current work and evidence in `TODO.md`. +3. Review and merge independently of distribution. A branch push or PR merge + does not publish a package, change repository visibility or create a tag. +4. Choose the release version after reviewing library/API compatibility, then + use the release steps below when explicitly authorized. Reconcile legacy + GitLab download/package links before announcing a Gitea release. + +Use the same Linux check script in Gitea CI when a Docker-capable runner is +configured. The current branch provides the local entry point; it does not +configure a runner or enable automatic publishing. + ## 1. Versioning and Tagging The project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). diff --git a/benches/benchmark.rs b/benches/benchmark.rs index 607f6fc..cee05ef 100644 --- a/benches/benchmark.rs +++ b/benches/benchmark.rs @@ -10,12 +10,12 @@ fn setup_test_dir() -> TempDir { // Create a moderately deep structure for i in 0..10 { - let dir_path = base_path.join(format!("dir_{}", i)); + let dir_path = base_path.join(format!("dir_{i}")); fs::create_dir(&dir_path).unwrap(); for j in 0..10 { - let file_path = dir_path.join(format!("file_{}.txt", j)); + let file_path = dir_path.join(format!("file_{j}.txt")); let mut file = File::create(file_path).unwrap(); - writeln!(file, "Some content for file {}-{}", i, j).unwrap(); + writeln!(file, "Some content for file {i}-{j}").unwrap(); } } temp_dir diff --git a/scripts/check-linux-container.sh b/scripts/check-linux-container.sh new file mode 100755 index 0000000..2bedbf4 --- /dev/null +++ b/scripts/check-linux-container.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail + +test "$(uname -s)" = Linux +test "$(id -u)" != 0 +printf 'Linux checks: uid=%s architecture=%s\n' "$(id -u)" "$(uname -m)" +rustc --version +cargo --version + +# Only source inputs are copied. Cargo output and all fixtures disappear with +# the container; the host checkout is read-only and no personal tree is scanned. +check_root=$(mktemp -d /tmp/sized-check.XXXXXX) +cp /source/Cargo.toml /source/Cargo.lock "$check_root/" +cp -R /source/src /source/tests /source/benches "$check_root/" +cd "$check_root" +cargo fmt --all -- --check +cargo test --locked +cargo test --locked --test linux_mount -- --ignored +cargo clippy --locked --all-targets -- -D warnings +cargo build --locked --release + +fixture_root=$(mktemp -d /tmp/sized-release.XXXXXX) +printf 'Linux release smoke test\n' > "$fixture_root/payload" +./target/release/sized --version +./target/release/sized "$fixture_root" --threads 2 --one-file-system --apparent --format json +printf 'Linux checks passed.\n' diff --git a/scripts/check-linux.sh b/scripts/check-linux.sh new file mode 100755 index 0000000..6768622 --- /dev/null +++ b/scripts/check-linux.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Pin the multi-platform official image, not a moving tag. Tests run on Linux +# filesystems rather than the source bind mount, whose permission semantics vary. +repo_root=$(cd "$(dirname "$0")/.." && pwd) +platform=${1:-linux/$(docker version --format '{{.Server.Arch}}')} +case "$platform" in + linux/arm64|linux/amd64) ;; + *) printf 'Usage: %s [linux/arm64|linux/amd64]\n' "$0" >&2; exit 2 ;; +esac +image=${SIZED_LINUX_IMAGE:-rust:1.88.0-bookworm@sha256:af306cfa71d987911a781c37b59d7d67d934f49684058f96cf72079c3626bfe0} +check_image="sized-linux-check:${platform#linux/}" +mkdir -p "$repo_root/target/linux-checks" +log_file="$repo_root/target/linux-checks/${platform#linux/}.log" + +docker build --platform "$platform" --build-arg "BASE_IMAGE=$image" \ + --tag "$check_image" - < "$repo_root/scripts/linux-check.Dockerfile" +docker run --rm --platform "$platform" \ + --user 65532:65532 --cap-drop ALL --security-opt no-new-privileges \ + --mount "type=bind,src=$repo_root,dst=/source,readonly" \ + --tmpfs /tmp/sized-mount-test:rw,nosuid,nodev,noexec,size=16m,uid=65532,gid=65532,mode=0700 \ + --tmpfs /tmp/sized-mount-test/foreign:rw,nosuid,nodev,noexec,size=16m,uid=65532,gid=65532,mode=0700 \ + --env CARGO_HOME=/tmp/sized-cargo \ + --env CARGO_BUILD_JOBS="${SIZED_LINUX_JOBS:-2}" \ + --env SIZED_TEST_MOUNT_ROOT=/tmp/sized-mount-test \ + "$check_image" bash /source/scripts/check-linux-container.sh 2>&1 | tee "$log_file" diff --git a/scripts/linux-check.Dockerfile b/scripts/linux-check.Dockerfile new file mode 100644 index 0000000..804ed5b --- /dev/null +++ b/scripts/linux-check.Dockerfile @@ -0,0 +1,3 @@ +ARG BASE_IMAGE=rust:1.88.0-bookworm@sha256:af306cfa71d987911a781c37b59d7d67d934f49684058f96cf72079c3626bfe0 +FROM ${BASE_IMAGE} +RUN rustup component add rustfmt clippy diff --git a/src/lib.rs b/src/lib.rs index 3925ba8..8f921ab 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -131,7 +131,7 @@ impl FromStr for SortColumn { "size" | "s" | "disk" | "d" => Ok(SortColumn::Disk), "apparent" | "a" => Ok(SortColumn::Apparent), "blocks" | "b" => Ok(SortColumn::Blocks), - _ => Err(format!("Unknown column: {}", s)), + _ => Err(format!("Unknown column: {s}")), } } } @@ -148,7 +148,7 @@ impl FromStr for SortDirection { match s.to_lowercase().as_str() { "asc" | "a" => Ok(SortDirection::Asc), "dsc" | "d" | "desc" => Ok(SortDirection::Dsc), - _ => Err(format!("Unknown direction: {}", s)), + _ => Err(format!("Unknown direction: {s}")), } } } @@ -182,7 +182,7 @@ pub fn run(args: Args, mut writer: &mut dyn Write) -> bool { match Byte::parse_str(size_str, true) { Ok(byte) => byte.as_u64(), Err(e) => { - eprintln!("Error parsing size '{}': {}", size_str, e); + eprintln!("Error parsing size '{size_str}': {e}"); std::process::exit(1); } } @@ -478,7 +478,7 @@ fn print_text( } } - writeln!(writer, "{}", summary).ok(); + writeln!(writer, "{summary}").ok(); } else { writeln!(writer, "Total size: {}", "Access Denied".bold().red()).ok(); } @@ -649,7 +649,7 @@ fn print_text( table.add_row(row); } } - writeln!(writer, "{}", table).ok(); + writeln!(writer, "{table}").ok(); } fn print_csv(writer: &mut dyn Write, children: &[&Node], args: &Args) { diff --git a/src/main.rs b/src/main.rs index a3a2edc..5edaec9 100644 --- a/src/main.rs +++ b/src/main.rs @@ -18,7 +18,7 @@ fn main() { OutputFormat::Json => "json", _ => "txt", }; - PathBuf::from(format!("{}_{}.{}", timestamp, dir_name, ext)) + PathBuf::from(format!("{timestamp}_{dir_name}.{ext}")) } else { path_arg.clone() }; diff --git a/src/scan.rs b/src/scan.rs index 8dd2ec1..bb6d0e8 100644 --- a/src/scan.rs +++ b/src/scan.rs @@ -58,7 +58,7 @@ pub enum EntryType { impl std::fmt::Display for EntryType { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "{:?}", self) + write!(f, "{self:?}") } } diff --git a/tests/linux_mount.rs b/tests/linux_mount.rs new file mode 100644 index 0000000..2e13f70 --- /dev/null +++ b/tests/linux_mount.rs @@ -0,0 +1,96 @@ +#![cfg(target_os = "linux")] + +use sized::{scan_tree, ScanControl, ScanOptions}; +use std::{fs, io::Write, os::unix::fs::MetadataExt, path::PathBuf, process::Command}; +use tempfile::NamedTempFile; + +#[test] +#[ignore = "requires the two owned tmpfs mounts from scripts/check-linux.sh"] +fn real_mount_boundary_is_respected_by_scanner_and_cli() { + let root = PathBuf::from( + std::env::var_os("SIZED_TEST_MOUNT_ROOT").expect("missing mounted Linux fixture"), + ); + let foreign = root.join("foreign"); + let root_metadata = fs::metadata(&root).unwrap(); + let foreign_metadata = fs::metadata(&foreign).unwrap(); + assert_ne!( + root_metadata.dev(), + foreign_metadata.dev(), + "not a real boundary" + ); + let mut local_file = NamedTempFile::new_in(&root).unwrap(); + let mut mounted_file = NamedTempFile::new_in(&foreign).unwrap(); + local_file.write_all(b"local allocation").unwrap(); + mounted_file.write_all(&[7; 8192]).unwrap(); + local_file.as_file().sync_all().unwrap(); + mounted_file.as_file().sync_all().unwrap(); + + for bounded in [false, true] { + let report = scan_tree( + &root, + &ScanOptions { + stay_on_filesystem: bounded, + threads: Some(2), + ..ScanOptions::default() + }, + &ScanControl::default(), + ) + .unwrap(); + assert!(report.root.complete); + assert!(report.issues.is_empty()); + let mount = report + .root + .children + .iter() + .find(|n| n.path == foreign) + .unwrap(); + assert_eq!(mount.identity.unwrap().device, foreign_metadata.dev()); + assert_eq!(mount.skipped_mount, bounded); + assert_eq!(report.entries_scanned, if bounded { 3 } else { 4 }); + let expected_blocks = + root_metadata.blocks() + local_file.as_file().metadata().unwrap().blocks(); + if bounded { + assert!(mount.children.is_empty()); + assert_eq!(mount.blocks, 0); + assert_eq!(mount.size_bytes, 0); + assert_eq!(report.root.blocks, expected_blocks); + } else { + assert_eq!(mount.children.len(), 1); + assert_eq!(mount.children[0].path, mounted_file.path()); + assert_eq!(mount.children[0].size_bytes, 8192); + assert_eq!( + report.root.blocks, + expected_blocks + + foreign_metadata.blocks() + + mounted_file.as_file().metadata().unwrap().blocks() + ); + } + + let mut command = Command::new(env!("CARGO_BIN_EXE_sized")); + command + .arg(&root) + .args(["--format", "json", "--apparent", "--threads", "2"]); + if bounded { + command.arg("--one-file-system"); + } + let output = command.output().unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert!(output.stderr.is_empty()); + let json: serde_json::Value = serde_json::from_slice(&output.stdout).unwrap(); + assert_eq!(json["complete"], true); + assert_eq!(json["total_blocks"], report.root.blocks); + let mount_json = json["entries"] + .as_array() + .unwrap() + .iter() + .find(|entry| entry["path"] == foreign.to_str().unwrap()) + .unwrap(); + assert_eq!(mount_json["skipped_mount"], bounded); + assert_eq!(mount_json["blocks"], mount.blocks); + assert_eq!(mount_json["apparent_size"], mount.size_bytes); + } +} -- 2.54.0 From a1b40d1826a92a15d589c3fb1bb8e890f614ac16 Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Fri, 9 Oct 2026 18:38:19 -0400 Subject: [PATCH 3/9] Record verified Linux and remote review checkpoint --- TODO.md | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 TODO.md diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..918622a --- /dev/null +++ b/TODO.md @@ -0,0 +1,90 @@ +# Live queue + +## Current scope + +Harden Sized for reuse by the native SpaceMonger-inspired SizeQueen project. +The user authorized fixes and updates discovered during that investigation. +Keep the existing CLI useful and preserve the GPL-3.0-only license. + +- [x] Correct allocated-size totals for hard links; retain apparent per-path + sizes and define deterministic ownership within each scan. +- [x] Preserve scan errors and expose incomplete results instead of silently + treating missing paths as accessible empty directories. +- [x] Expose cancellable scans, progress counters, per-scan worker selection, + and an optional filesystem boundary for the native UI. +- [x] Add accounting and control regressions; run the existing CLI tests and + compare bounded release-build scan measurements. +- [x] Run reproducible Linux ARM64 and AMD64 Docker checks as an unprivileged + user, including actual mount boundaries, strict Clippy and a release smoke test. +- [x] Commit the scanner hardening and Linux check tooling; push the existing + `sizequeen-scan-hardening` review branch. Remote equality is verified. + Open a PR when ready; release/tag/package publication remains separate. + +## Proposed next work + +SizeQueen now includes a byte-identical scanner snapshot in a small core crate, +removing terminal dependencies and allowing independent builds. Extract that +shared core upstream in a separate PR after the behaviour changes are reviewed; +terminal formatting is already separate from scan logic. +Windows allocation support and very-large/deep-tree tuning need separate evidence. +README/MANUAL/Cargo metadata still contain legacy GitLab addresses; reconcile +them against available Gitea releases before changing distribution instructions. +Before producing release assets, fix and verify `scripts/release.sh`: it currently +creates the tarball before copying the executable into its input directory. +Do not release/tag/publish automatically. The requested review-branch push is +authorized; see `SHIPMENT.md` for source review and the separate release process. + +## Resume note + +Linux validation and the requested remote review branch are complete. +Scanner hardening is `8b177e2`; repeatable Linux checks and equivalent format +interpolations are `988aad9`. Both implementation commits are pushed to +`origin/sizequeen-scan-hardening`; SSH verified remote/local equality at +`988aad99510c0424cf0db5a116d9086183da74a5`. Gitea main remains `9c8d1d4`. +The repository was already public (`private: false`); visibility is unchanged. +No PR, merge, tag, package publication or release occurred. + +`scripts/check-linux.sh` owns the container workflow. Linux ARM64 (native in +Docker's VM) and AMD64 (emulated on this Mac) each passed 23 tests as UID 65532: +5 library, 6 CLI, 11 scan integrations and one explicitly enabled real-mount +integration. `tests/linux_mount.rs` checks two distinct tmpfs devices through +the library and CLI, with boundaries enabled and disabled. Formatting, strict +all-target Clippy and an optimized-binary fixture smoke test passed on both. +Mac passed its 22 applicable tests, formatting and strict Clippy. Commands are +documented in README/SHIPMENT; full local logs are in ignored +`target/linux-checks/{arm64,amd64}.log`. Cargo.lock and unrelated `.DS_Store` +hashes are unchanged; only source inputs and owned fixtures enter the checks. + +Rust 1.88 Clippy identified format-string style warnings; equivalent +interpolations fix those without suppressions. SizeQueen's included scanner +still matches `8b177e2` exactly; the follow-up changes only one scanner Display +format string, not scan behaviour. SizeQueen itself was unchanged in this pass. +Next: open/review a PR against main, calling out CLI partial-scan status and +library/API changes. Keep shared-core extraction as the following proposed PR, +then choose a release version and repair/verify packaging when release work +is authorized. These checks do not prove desktop X11/Wayland interaction, +Windows allocation or performance on very large/cold/remote trees. + +Previous local checkpoint: baseline `9c8d1d4` matched Gitea main; +branch `sizequeen-scan-hardening`. +All six baseline tests passed. SizeQueen's independent probe reproduced +hard-link overcount and missing-path misclassification, and confirmed sparse +allocation, symlink leaf handling, and hidden-file inclusion. Source inspection +found discarded walker errors and no scan control API. These scoped corrections +are implemented locally, with no push, tag, or release. `cargo test --locked` +passed 22 tests (5 library, 6 CLI, 11 scan integrations); strict all-target +Clippy passed. The independent SizeQueen probe passed 6 accounting tests. +Man page was regenerated. Cargo.lock is unchanged and unrelated `.DS_Store` +was preserved. + +Release probes on the Mac took 49–56ms for 20,000 files; sorting, identity +tracking, diagnostics and control cost more than the baseline (29–39ms for +the grouped tree). Inline Node size grew from 88 to 136 bytes; whole-process +peak RSS was about 8.2MiB grouped / 16.7MiB wide. These are bounded warm-metadata +fixtures, not evidence for million-entry, cold-disk, or remote-filesystem speed. +Cancellation is cooperative between filesystem calls. Allocation is reported +blocks, not guaranteed bytes recoverable from shared extents or snapshots. + +Detailed audit and bounded probes are maintained in the sibling SizeQueen +project at `../sizequeen/research/SIZED-AUDIT.md`; that project's product queue +remains separate from this backend's fix queue. -- 2.54.0 From 67fe175ec728fd13303eb982e16a8ab2920cc434 Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Fri, 9 Oct 2026 18:46:33 -0400 Subject: [PATCH 4/9] Run Sized Linux checks on the existing Gitea runner --- .gitea/workflows/linux.yml | 40 ++++++++++++++++++++++++++++++++ README.md | 7 ++++++ SHIPMENT.md | 27 ++++++++++++++++++--- TODO.md | 11 +++++++++ scripts/check-linux-container.sh | 5 ++-- scripts/gitea-linux.Dockerfile | 10 ++++++++ 6 files changed, 95 insertions(+), 5 deletions(-) create mode 100644 .gitea/workflows/linux.yml create mode 100644 scripts/gitea-linux.Dockerfile diff --git a/.gitea/workflows/linux.yml b/.gitea/workflows/linux.yml new file mode 100644 index 0000000..49d35b0 --- /dev/null +++ b/.gitea/workflows/linux.yml @@ -0,0 +1,40 @@ +name: Sized Linux checks + +on: + push: + branches: [main, sizequeen-scan-hardening] + paths: ['Cargo.toml', 'Cargo.lock', 'src/**', 'tests/**', 'benches/**', 'scripts/**', '.gitea/workflows/**'] + pull_request: + branches: [main] + paths: ['Cargo.toml', 'Cargo.lock', 'src/**', 'tests/**', 'benches/**', 'scripts/**', '.gitea/workflows/**'] + workflow_dispatch: + +permissions: + contents: read + +jobs: + linux-amd64: + name: Linux AMD64 / Rust 1.88.0 + # Keep the existing shared runner limited to owner-triggered, same-repo code. + if: ${{ gitea.actor == 'gamertan' && (gitea.event_name != 'pull_request' || gitea.event.pull_request.head.repo.full_name == gitea.repository) }} + runs-on: himesan-node24 + timeout-minutes: 20 + container: + image: sha256:514512270649a85769c2b8ecfcc3a860d3a7da67c29b9b7b1cfe4c75de41d322 + options: >- + --user 65532:65532 + --tmpfs /tmp/sized-mount-test:rw,nosuid,nodev,noexec,size=16m,uid=65532,gid=65532,mode=0700 + --tmpfs /tmp/sized-mount-test/foreign:rw,nosuid,nodev,noexec,size=16m,uid=65532,gid=65532,mode=0700 + env: + CARGO_HOME: /tmp/sized-cargo + CARGO_BUILD_JOBS: '2' + SIZED_TEST_SOURCE: ${{ gitea.workspace }} + SIZED_TEST_MOUNT_ROOT: /tmp/sized-mount-test + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 + with: + persist-credentials: false + - name: Unprivileged Linux tests, real mount boundary, Clippy and release smoke + run: ./scripts/check-linux-container.sh + - name: Require an unchanged checkout + run: test -z "$(git status --porcelain=v1 --untracked-files=all)" diff --git a/README.md b/README.md index cedfbfc..fd00c94 100644 --- a/README.md +++ b/README.md @@ -170,6 +170,13 @@ removed on exit. Docker retains the reusable check images/build cache. can select a different toolchain image for an explicit compatibility check. These are backend checks; desktop X11/Wayland acceptance belongs to SizeQueen. +Gitea's `Sized Linux checks` workflow runs the same checks natively on cliff-mads +for main/review-branch source changes and owner-triggered, same-repository PRs. +It uses a pinned Rust job image, UID 65532 and real tmpfs boundaries without +Docker access inside the job. Fork contributions can run the local script; +maintainers can bring reviewed changes onto a repository branch for CI. +See SHIPMENT for runner-image setup. Documentation-only pushes skip builds. + See [source review and release process](SHIPMENT.md) and the [live queue](TODO.md). ## License diff --git a/SHIPMENT.md b/SHIPMENT.md index fdd2646..450d073 100644 --- a/SHIPMENT.md +++ b/SHIPMENT.md @@ -23,9 +23,30 @@ in a separate PR so reviewers can distinguish behaviour changes from packaging. use the release steps below when explicitly authorized. Reconcile legacy GitLab download/package links before announcing a Gitea release. -Use the same Linux check script in Gitea CI when a Docker-capable runner is -configured. The current branch provides the local entry point; it does not -configure a runner or enable automatic publishing. +### Gitea Linux CI + +`.gitea/workflows/linux.yml` uses the existing `himesan-node24` label on +cliff-mads, overriding the job image with the pinned Sized Rust runtime. +The runner itself launches the two tmpfs mounts. Jobs have no Docker socket +and run `scripts/check-linux-container.sh` as UID 65532, using the checked-out +workspace as `SIZED_TEST_SOURCE`. Its tests, strict Clippy, formatting and +optimized-binary smoke test are the same as the local Docker workflow. + +The runtime contains Rust 1.88.0, rustfmt/Clippy and Node for the pinned checkout +action. On cliff-mads, rebuild it explicitly with: + +```bash +ssh cliff-mads 'docker build --platform linux/amd64 --tag local/gamertan-ci:sized-rust188 -' < scripts/gitea-linux.Dockerfile +ssh cliff-mads 'docker image inspect local/gamertan-ci:sized-rust188 --format "{{.Id}}"' +``` + +Review and update the workflow's image ID after an intentional rebuild; images +stay local to the runner host. No runner labels, global isolation settings or +publishing credentials are required. Repository Actions must be enabled. +Only owner-triggered, same-repository code runs on this shared homelab runner. +Source changes on main/the review branch and PRs to main trigger checks; +documentation-only pushes skip builds. Manual dispatch is also available. +This workflow does not publish or tag releases. ## 1. Versioning and Tagging diff --git a/TODO.md b/TODO.md index 918622a..876d328 100644 --- a/TODO.md +++ b/TODO.md @@ -19,6 +19,9 @@ Keep the existing CLI useful and preserve the GPL-3.0-only license. - [x] Commit the scanner hardening and Linux check tooling; push the existing `sizequeen-scan-hardening` review branch. Remote equality is verified. Open a PR when ready; release/tag/package publication remains separate. +- [ ] Enable repository Actions and run the same Linux checks on the existing + cliff-mads runner. Keep its container isolation and other jobs unchanged; + verify a real workflow result before treating CI as proven. ## Proposed next work @@ -36,6 +39,14 @@ authorized; see `SHIPMENT.md` for source review and the separate release process ## Resume note +Current CI checkpoint: repository Actions is enabled. The existing cliff-mads +runner is healthy (`gitea-runner v3.1.0`, native AMD64). The workflow reuses its +`himesan-node24` label and a separately built Rust/Node image; runner configuration, +other jobs and repository visibility are unchanged. The image bootstrap probe +verified UID 65532, workspace-volume ownership, Rust 1.88.0/Node 24.19.0 and +distinct tmpfs devices. Actual Gitea checkout/check execution is still pending; +do not call CI proven until the real workflow completes. + Linux validation and the requested remote review branch are complete. Scanner hardening is `8b177e2`; repeatable Linux checks and equivalent format interpolations are `988aad9`. Both implementation commits are pushed to diff --git a/scripts/check-linux-container.sh b/scripts/check-linux-container.sh index 2bedbf4..a8b3544 100755 --- a/scripts/check-linux-container.sh +++ b/scripts/check-linux-container.sh @@ -10,8 +10,9 @@ cargo --version # Only source inputs are copied. Cargo output and all fixtures disappear with # the container; the host checkout is read-only and no personal tree is scanned. check_root=$(mktemp -d /tmp/sized-check.XXXXXX) -cp /source/Cargo.toml /source/Cargo.lock "$check_root/" -cp -R /source/src /source/tests /source/benches "$check_root/" +source_root=${SIZED_TEST_SOURCE:-/source} +cp "$source_root/Cargo.toml" "$source_root/Cargo.lock" "$check_root/" +cp -R "$source_root/src" "$source_root/tests" "$source_root/benches" "$check_root/" cd "$check_root" cargo fmt --all -- --check cargo test --locked diff --git a/scripts/gitea-linux.Dockerfile b/scripts/gitea-linux.Dockerfile new file mode 100644 index 0000000..26572b9 --- /dev/null +++ b/scripts/gitea-linux.Dockerfile @@ -0,0 +1,10 @@ +# Node supports the pinned checkout action; application code remains Rust. +FROM node:24-bookworm@sha256:934240a162082fd8b8a2f90cd5114446443f1eba1c5378f6687167ca405e6584 AS node-runtime +FROM rust:1.88.0-bookworm@sha256:af306cfa71d987911a781c37b59d7d67d934f49684058f96cf72079c3626bfe0 +RUN rustup component add rustfmt clippy +COPY --from=node-runtime /usr/local/bin/node /usr/local/bin/node +# A new runner workspace volume inherits this ownership. No setuid step or +# Docker socket is needed in jobs, even with all capabilities dropped. +RUN mkdir -p /workspace && chown 65532:65532 /workspace +USER 65532:65532 +WORKDIR /workspace -- 2.54.0 From 74b30bbf750417121f3b0c26017d2f011a2a8286 Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Fri, 9 Oct 2026 18:47:58 -0400 Subject: [PATCH 5/9] Prepare repository workspace for unprivileged Gitea checkout --- .gitea/workflows/linux.yml | 2 +- SHIPMENT.md | 4 +++- scripts/gitea-linux.Dockerfile | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.gitea/workflows/linux.yml b/.gitea/workflows/linux.yml index 49d35b0..5c72148 100644 --- a/.gitea/workflows/linux.yml +++ b/.gitea/workflows/linux.yml @@ -20,7 +20,7 @@ jobs: runs-on: himesan-node24 timeout-minutes: 20 container: - image: sha256:514512270649a85769c2b8ecfcc3a860d3a7da67c29b9b7b1cfe4c75de41d322 + image: sha256:271ca1d26057c95a6fd30df7ccc0c053ab394c0e81cd1e4efa182b5b0b60ee97 options: >- --user 65532:65532 --tmpfs /tmp/sized-mount-test:rw,nosuid,nodev,noexec,size=16m,uid=65532,gid=65532,mode=0700 diff --git a/SHIPMENT.md b/SHIPMENT.md index 450d073..6dafe0c 100644 --- a/SHIPMENT.md +++ b/SHIPMENT.md @@ -41,7 +41,9 @@ ssh cliff-mads 'docker image inspect local/gamertan-ci:sized-rust188 --format "{ ``` Review and update the workflow's image ID after an intentional rebuild; images -stay local to the runner host. No runner labels, global isolation settings or +stay local to the runner host. The image pre-owns `/workspace/gamertan/sized` +for UID 65532 so the runner's workspace setup permits unprivileged checkout. +No runner labels, global isolation settings or publishing credentials are required. Repository Actions must be enabled. Only owner-triggered, same-repository code runs on this shared homelab runner. Source changes on main/the review branch and PRs to main trigger checks; diff --git a/scripts/gitea-linux.Dockerfile b/scripts/gitea-linux.Dockerfile index 26572b9..b0f95ac 100644 --- a/scripts/gitea-linux.Dockerfile +++ b/scripts/gitea-linux.Dockerfile @@ -5,6 +5,6 @@ RUN rustup component add rustfmt clippy COPY --from=node-runtime /usr/local/bin/node /usr/local/bin/node # A new runner workspace volume inherits this ownership. No setuid step or # Docker socket is needed in jobs, even with all capabilities dropped. -RUN mkdir -p /workspace && chown 65532:65532 /workspace +RUN mkdir -p /workspace/gamertan/sized && chown -R 65532:65532 /workspace USER 65532:65532 WORKDIR /workspace -- 2.54.0 From 9fd4e905b00ad23cea06bfaa1cc6a9608032266c Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Fri, 9 Oct 2026 18:53:03 -0400 Subject: [PATCH 6/9] Record successful cliff-mads Linux CI checkpoint --- SHIPMENT.md | 7 +++++++ TODO.md | 28 ++++++++++++++++++---------- 2 files changed, 25 insertions(+), 10 deletions(-) diff --git a/SHIPMENT.md b/SHIPMENT.md index 6dafe0c..dd37a1e 100644 --- a/SHIPMENT.md +++ b/SHIPMENT.md @@ -50,6 +50,13 @@ Source changes on main/the review branch and PRs to main trigger checks; documentation-only pushes skip builds. Manual dispatch is also available. This workflow does not publish or tag releases. +The first complete native AMD64 push check is +[Gitea run 1048 (number 2)](https://gitea.speelman.ca/gamertan/sized/actions/runs/1048), +at `74b30bbf750417121f3b0c26017d2f011a2a8286`. All 23 tests, formatting, strict +Clippy, release smoke and unchanged-checkout verification passed as UID 65532 +on `cliff-himesan-linux-amd64`. PR and manual-dispatch events are configured; +only push execution has been exercised. See `TODO.md` for the current checkpoint. + ## 1. Versioning and Tagging The project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). diff --git a/TODO.md b/TODO.md index 876d328..bd60091 100644 --- a/TODO.md +++ b/TODO.md @@ -19,7 +19,7 @@ Keep the existing CLI useful and preserve the GPL-3.0-only license. - [x] Commit the scanner hardening and Linux check tooling; push the existing `sizequeen-scan-hardening` review branch. Remote equality is verified. Open a PR when ready; release/tag/package publication remains separate. -- [ ] Enable repository Actions and run the same Linux checks on the existing +- [x] Enable repository Actions and run the same Linux checks on the existing cliff-mads runner. Keep its container isolation and other jobs unchanged; verify a real workflow result before treating CI as proven. @@ -39,19 +39,27 @@ authorized; see `SHIPMENT.md` for source review and the separate release process ## Resume note -Current CI checkpoint: repository Actions is enabled. The existing cliff-mads -runner is healthy (`gitea-runner v3.1.0`, native AMD64). The workflow reuses its -`himesan-node24` label and a separately built Rust/Node image; runner configuration, -other jobs and repository visibility are unchanged. The image bootstrap probe -verified UID 65532, workspace-volume ownership, Rust 1.88.0/Node 24.19.0 and -distinct tmpfs devices. Actual Gitea checkout/check execution is still pending; -do not call CI proven until the real workflow completes. +Current CI checkpoint: repository Actions is enabled and a real native AMD64 +push run passed on cliff-mads. [Gitea run 1048 (number 2)](https://gitea.speelman.ca/gamertan/sized/actions/runs/1048) +tested `74b30bbf750417121f3b0c26017d2f011a2a8286` on +`cliff-himesan-linux-amd64` (`gitea-runner v3.1.0`): all 23 tests, formatting, +strict Clippy, optimized-binary smoke and unchanged-checkout verification passed +as UID 65532 with Rust 1.88.0. It finished in 3m 30s. Filtered local evidence is +in ignored `target/linux-checks/gitea-run-1048.log`; full logs remain in Gitea. +The initial checkout failed because the repository directory was root-owned; +the corrected runtime pre-owns it for the job user. The workflow reuses the +existing `himesan-node24` label and a local, pinned Rust/Node image. Runner +configuration, other jobs and repository visibility are unchanged. Temporary +setup credentials were revoked and their files removed. Push execution is +proven; PR/manual-dispatch triggers are configured but not independently run. Linux validation and the requested remote review branch are complete. Scanner hardening is `8b177e2`; repeatable Linux checks and equivalent format interpolations are `988aad9`. Both implementation commits are pushed to -`origin/sizequeen-scan-hardening`; SSH verified remote/local equality at -`988aad99510c0424cf0db5a116d9086183da74a5`. Gitea main remains `9c8d1d4`. +`origin/sizequeen-scan-hardening`, followed by CI setup `67fe175` and the +unprivileged checkout fix `74b30bb`. The source checkpoint is verified remotely +at `74b30bbf750417121f3b0c26017d2f011a2a8286`; later documentation-only +checkpoint commits do not rerun the source checks. Gitea main remains `9c8d1d4`. The repository was already public (`private: false`); visibility is unchanged. No PR, merge, tag, package publication or release occurred. -- 2.54.0 From 5f0b11ea2efb72d361a282ed9bb65003a889e299 Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Sat, 10 Oct 2026 03:51:35 -0400 Subject: [PATCH 7/9] Repair release archives and document current Sized distribution --- Cargo.toml | 3 +- MANUAL.md | 22 +++++-- Makefile | 2 +- README.md | 103 +++++++++++++++---------------- SHIPMENT.md | 69 +++++++++++++-------- TODO.md | 44 +++++++++---- scripts/check-linux-container.sh | 5 ++ scripts/check-release.sh | 32 ++++++++++ scripts/release.sh | 101 +++++++++++++++--------------- 9 files changed, 233 insertions(+), 148 deletions(-) create mode 100755 scripts/check-release.sh diff --git a/Cargo.toml b/Cargo.toml index 8fb825e..ec8a6c8 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,7 +4,8 @@ version = "1.0.0" edition = "2021" description = "A modern, fast, and concurrent disk usage analyzer" license = "GPL-3.0-only" -repository = "https://gitlab.speelman.ca/gamertan/sized" +repository = "https://gitea.speelman.ca/gamertan/sized" +homepage = "https://gamertan.com/projects/sized/" readme = "README.md" categories = ["command-line-utilities", "filesystem"] diff --git a/MANUAL.md b/MANUAL.md index 8143987..92e0170 100644 --- a/MANUAL.md +++ b/MANUAL.md @@ -16,16 +16,28 @@ ## Installation -### From Binaries (Recommended) -Download the latest pre-compiled binaries from the [Releases](https://gitlab.speelman.ca/gamertan/sized/releases) page. +### From source + +The following installs the current review branch, including scanner hardening +not present in the published v1.0.0 release: -### From Source ```bash -git clone https://gitlab.speelman.ca/gamertan/sized.git +git clone --branch sizequeen-scan-hardening https://gitea.speelman.ca/gamertan/sized.git cd sized -make install # Installs binary and man page +cargo install --locked --path . ``` +Cargo installs to `~/.cargo/bin`. For the published source instead, check out +`v1.0.0` before installing. `make install PREFIX="$HOME/.local"` additionally +installs the man page without requiring administrator privileges. + +### Existing release files + +The [v1.0.0 release](https://gitea.speelman.ca/gamertan/sized/releases/tag/v1.0.0) +has source archives, a macOS arm64 executable, a man page and completions. +It has no Linux binary, architecture-labelled binary archive or `.deb` attachment. +See the [README](README.md#installation) for current installation guidance. + ## Basic Usage By default, `sized` analyzes the current directory recursively and displays a table of the immediate children. diff --git a/Makefile b/Makefile index a5c8057..5f7d326 100644 --- a/Makefile +++ b/Makefile @@ -7,7 +7,7 @@ MANDIR = $(PREFIX)/share/man/man1 all: build build: - cargo build --release + cargo build --locked --release install: build install -d $(BINDIR) diff --git a/README.md b/README.md index fd00c94..59cad2d 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,20 @@ # Sized -A fast, concurrent, and feature-rich command-line tool for visualizing disk usage, written in Rust. +**Know what’s taking up space. Without leaving your terminal.** + +Sized is a Rust command-line disk usage tool for macOS and Linux. Inspect large +folders, filter the noise, and export reports for your own scripts. +[Project page](https://gamertan.com/projects/sized/) · +[Prefer a visual map? Meet SizeQueen](https://gamertan.com/projects/sizequeen/). + +This branch contains scanner hardening under review. The published **v1.0.0** +release predates the hard-link, partial-scan and filesystem-boundary changes +below. No new release is implied by a branch build. ## Features - **Allocation by Default**: Prioritizes filesystem-reported allocated blocks; sparse files retain their separate apparent size, and hard-link allocation is counted once per scan. Reported allocation is not a promise of bytes freed by deletion on filesystems with shared extents or snapshots. -- **Fast & Concurrent**: Uses `rayon` to process directories in parallel, making it extremely fast on modern multi-core systems. +- **Fast & Concurrent**: Processes directories in parallel with `rayon`; scan time depends on the filesystem and workload. - **Rich Output**: Beautifully formatted tables with colors, distinguishing files and directories. - **Apparent Size**: Toggle logical file length with `-a` or `--apparent`. - **Unit Selection**: Switch between Binary (IEC) and Decimal (SI) unit systems via `--units`. @@ -18,56 +27,40 @@ A fast, concurrent, and feature-rich command-line tool for visualizing disk usag ## Installation -### 🚀 Direct Download (macOS & Linux) -Download the latest archive for your architecture from the [Releases](https://gitlab.speelman.ca/gamertan/sized/releases) page. +### Build the current review branch -1. **Extract the archive**: - ```bash - tar -xzf sized-v1.0.0-Darwin-arm64.tar.gz - ``` +With a Rust toolchain and Git installed: -2. **Install Binary & Man Page**: - ```bash - # Move binary to path - sudo mv sized /usr/local/bin/ - - # Install man page - sudo mkdir -p /usr/local/share/man/man1 - sudo cp sized.1 /usr/local/share/man/man1/ - ``` - -3. **macOS Security (Gatekeeper)**: - Since the binary isn't code-signed for the App Store, macOS may block it. To allow it: - ```bash - sudo xattr -d com.apple.quarantine /usr/local/bin/sized - ``` - *Alternatively, run `sized` once, let it fail, then go to **System Settings > Privacy & Security** and click **"Allow Anyway"**.* - -### 🍺 Homebrew (macOS & Linux) -If you have a homebrew tap: ```bash -brew install gamertan/tap/sized -``` - -### 📦 Debian / Ubuntu (.deb) -1. Download the `.deb` package from the [Releases](https://gitlab.speelman.ca/gamertan/sized/releases) page. -2. Install using `dpkg`: - ```bash - sudo dpkg -i sized_1.0.0_amd64.deb - ``` - -### 🦀 From Source (Rust toolchain required) -```bash -git clone https://gitlab.speelman.ca/gamertan/sized.git +git clone --branch sizequeen-scan-hardening https://gitea.speelman.ca/gamertan/sized.git cd sized -make install # Installs binary and man page +cargo install --locked --path . +sized --help ``` -Alternatively, via Cargo: +Cargo installs into `~/.cargo/bin`; include it in your `PATH`. For the published +source instead, check out `v1.0.0` before the install command. Source builds run +locally; they are separate from SizeQueen's signed and notarized Mac app. + +To install the binary and man page together without administrator privileges: + ```bash -cargo install --path . +make install PREFIX="$HOME/.local" ``` +This uses `~/.local/bin` and `~/.local/share/man/man1`; configure `PATH` and your +manual-page search path as needed. + +### Existing release downloads + +The [v1.0.0 release](https://gitea.speelman.ca/gamertan/sized/releases/tag/v1.0.0) +contains source, one legacy executable named `sized`, a man page and shell +completions. The executable was inspected as macOS arm64 (Apple silicon); it is not a Linux download. +There are currently no attached architecture-labelled archives or `.deb` +packages, and no verified Homebrew tap. Build from source for the current +review changes. Platform-labelled archives will be advertised after a new +release is reviewed and published. + ## Documentation - **[Manual](MANUAL.md)**: Detailed explanations of all flags and features. - **[Man Page](sized.1)**: Standard unix man pages (installed via `make install`). @@ -177,19 +170,25 @@ Docker access inside the job. Fork contributions can run the local script; maintainers can bring reviewed changes onto a repository branch for CI. See SHIPMENT for runner-image setup. Documentation-only pushes skip builds. +Run `./scripts/check-release.sh` to verify the actual archive, checksum and +extracted executable on the host. Linux CI also runs this packaging check. + See [source review and release process](SHIPMENT.md) and the [live queue](TODO.md). +## Sized, SizeQueen and the shared scanner + +Sized began as terminal tooling. Its filesystem scanner was extracted into +`sized-core`, which SizeQueen now uses directly beneath its native Mac interface. +SizeQueen adds the treemap and desktop interactions; it does not run the CLI. +Sized currently retains its own scanner copy while shared-core adoption is +reviewed. A public checkout of Sized does not need access to the private core +repository. SizeQueen's matching source download includes its pinned core. + ## License -This project is licensed under the **GNU General Public License v3.0 (GPL-3.0)**. - -### Why GPL-3.0? (The "Insulin" Philosophy) -We believe that core diagnostic tools like `sized` should remain a public good. Inspired by the philosophy behind open-access medicine like insulin, this license ensures that: -- The tool remains **free and open** for everyone to use. -- Any improvements or forks made by others **must also be shared** with the community. -- It prevents proprietary "vampire" versions from taking private credit for public efforts. - -For more details, see the [LICENSE](LICENSE) file. +**GPL-3.0-only.** See [LICENSE](LICENSE) for the complete terms. All Sized features +are free; optional [support for Gamertan](https://gamertan.com/store/) does not +unlock features. ## Contributing diff --git a/SHIPMENT.md b/SHIPMENT.md index dd37a1e..e24ea0e 100644 --- a/SHIPMENT.md +++ b/SHIPMENT.md @@ -6,8 +6,9 @@ This document defines the process for versioning and distributing the `sized` pr Use a named branch from `main` and keep a pull request focused on one outcome. The `sizequeen-scan-hardening` branch corrects accounting/error handling and -adds the scan controls needed by SizeQueen; shared-crate extraction follows -in a separate PR so reviewers can distinguish behaviour changes from packaging. +adds the scan controls needed by SizeQueen; the scanner has also been +extracted into private `sized-core` for SizeQueen. Sized retains its own copy +until adoption preserves public source access. 1. Run local locked tests and strict Clippy as an unprivileged user. Run `./scripts/check-linux.sh linux/arm64` and @@ -20,8 +21,8 @@ in a separate PR so reviewers can distinguish behaviour changes from packaging. 3. Review and merge independently of distribution. A branch push or PR merge does not publish a package, change repository visibility or create a tag. 4. Choose the release version after reviewing library/API compatibility, then - use the release steps below when explicitly authorized. Reconcile legacy - GitLab download/package links before announcing a Gitea release. + use the release steps below when explicitly authorized. Verify the exact archive + contents, target and installation instructions before announcing a release. ### Gitea Linux CI @@ -68,40 +69,54 @@ The project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html git tag -a v1.0.0 -m "Release v1.0.0" ``` -## 2. Asset Generation - -The `scripts/release.sh` script is the authoritative way to generate release assets locally or in any CI environment. +## 2. Asset generation and verification ```bash +./scripts/check-release.sh ./scripts/release.sh ``` -This script generates: -- **Optimized Binary**: The production-ready `sized` executable. -- **Documentation**: The `sized.1` man page. -- **Shell Completions**: Scripts for Bash, Zsh, and Fish. -- **System Installers**: A `.deb` package for Debian-based Linux distributions. +`release.sh` uses the locked dependencies and the Rust toolchain's host target. +It packages the executable **before** creating the archive, together with the +man page, Bash/Zsh/Fish completions, licence, README, manual and build metadata. +Outputs are `dist/sized-v--/`, a sibling `.tar.gz` +and `.tar.gz.sha256`. Existing outputs are never replaced. Use +`--output-dir /absolute/new/path` for an explicitly named local candidate. +`CARGO_TARGET_DIR` is respected. The script does not sign, upload, tag or publish. -All assets are gathered in the `dist/v/` directory. +`check-release.sh` creates temporary outputs, extracts the archive, checks all +required files and the checksum, then runs the packaged executable against an +owned fixture. It also checks the refusal to overwrite. Gitea's Linux check +runs this same test; a successful compile alone is insufficient. -## 3. Distribution Channels +Only build from a clean, reviewed commit for distribution. `BUILD-INFO.txt` +records the commit, target, toolchain and whether tracked inputs were modified. +Exported source can provide `SIZED_SOURCE_REVISION`; verify that provenance +against the original checkout. Retain matching source and complete dependency +licences with any public binary distribution. A host build is unsigned; do not +advertise it as notarized or as a Mac App Store application. -### crates.io -To update the project on the central Rust registry: -```bash -cargo publish -``` +## 3. Distribution channels -### System Package Managers -For Homebrew, GitLab, or Gitea-based distribution: -1. Push the git tag to your SCM. -2. Run the release script to generate assets. -3. Attach the contents of the `dist/` folder to your SCM's "Release" or "Tag" entry. -4. Update downstream formulas (like `homebrew-tap`) with the new source URL and SHA256 of the generated tarball. +The existing public Gitea v1.0.0 release contains a macOS arm64 executable, +man page and completions, plus generated source archives. It predates the +scanner hardening branch. No architecture-labelled binary archive, `.deb` or +verified Homebrew tap is currently offered. -## 4. Automation & Hooks +Future public releases need their own reviewed version, source, target-specific +archives, checksums and installation acceptance. Do not replace historical +v1.0.0 assets with newly built bytes. Add a new version only after review. -To automate asset generation locally, you can use a git `post-checkout` or a wrapper script. Since gitea and gitlab use different CI formats, it is recommended to simply call `./scripts/release.sh` within your preferred CI runner (e.g., `gitlab-ci.yml` or `gitea-actions`). +Debian/Alpine packaging is separate from the portable host archive. Do not +implicitly invoke `cargo deb` or `abuild` just because they are installed: a Mac +binary is not a Debian package. Validate each installer on its target OS before +publishing it. Registry and Homebrew publication are separate decisions too. + +## 4. Automation + +CI tests and packages temporary candidates without publishing credentials. +Do not run release generation automatically from Git hooks. Publication remains +an intentional operation after the checks above. ## 5. Manual Installation For system-wide installation from source, use the `Makefile`: diff --git a/TODO.md b/TODO.md index bd60091..c09085c 100644 --- a/TODO.md +++ b/TODO.md @@ -23,23 +23,41 @@ Keep the existing CLI useful and preserve the GPL-3.0-only license. cliff-mads runner. Keep its container isolation and other jobs unchanged; verify a real workflow result before treating CI as proven. -## Proposed next work +## Approved project page and release cleanup -SizeQueen now includes a byte-identical scanner snapshot in a small core crate, -removing terminal dependencies and allowing independent builds. Extract that -shared core upstream in a separate PR after the behaviour changes are reviewed; -terminal formatting is already separate from scan logic. -Windows allocation support and very-large/deep-tree tuning need separate evidence. -README/MANUAL/Cargo metadata still contain legacy GitLab addresses; reconcile -them against available Gitea releases before changing distribution instructions. -Before producing release assets, fix and verify `scripts/release.sh`: it currently -creates the tarball before copying the executable into its input directory. -Do not release/tag/publish automatically. The requested review-branch push is -authorized; see `SHIPMENT.md` for source review and the separate release process. +The owner approved a Sized project page in the shared Gamertan CMS, truthful +installation/release guidance, packaging repairs, and a Built on Sized section +on SizeQueen. Website delivery is tracked in that repository's existing queue. + +- [x] Inspect the public v1.0.0 assets; the unlabelled executable is macOS arm64, + ad hoc signed, SHA-256 `50fde4d8215babbb64f4a4f55e030dd5c941a97ed1bedfa80392e4301c52d2bf`. + There are no attached Linux binaries, labelled archives or Debian packages. +- [x] Replace stale GitLab/unsupported package-manager installation claims with + explicit review-branch source builds; explain the published release boundary. +- [x] Package the executable before the archive; include licence/docs/build + provenance, target-labelled names and checksums. Reject output replacement. + Remove incidental installer generation; host binaries must not become `.deb`s. +- [x] Verify archive contents/checksum, run the extracted CLI and test overwrite + refusal on macOS arm64. Add the same verification to Linux CI. +- [ ] Verify the updated cliff-mads workflow, push the review checkpoint and + record the public CMS delivery. No new version, release assets or merge yet. + +## Deferred / next release + +SizeQueen now pins the extracted private `sized-core`; Sized retains its own +scanner copy. Adoption must keep public builds independently fetchable. +Review scanner/API/status changes before merging the existing branch. Select a +new release version, verify target-specific installation and package complete +corresponding source/dependency notices before publishing fresh binary assets. +Do not replace historical v1.0.0 files. Windows allocation and very-large/deep +scan tuning remain separate work. See `SHIPMENT.md` for the release process. ## Resume note -Current CI checkpoint: repository Actions is enabled and a real native AMD64 +Current work: the approved project page and packaging cleanup above. Local Mac +archive verification passes; updated Linux CI and CMS publication are pending. + +Previous CI checkpoint: repository Actions is enabled and a real native AMD64 push run passed on cliff-mads. [Gitea run 1048 (number 2)](https://gitea.speelman.ca/gamertan/sized/actions/runs/1048) tested `74b30bbf750417121f3b0c26017d2f011a2a8286` on `cliff-himesan-linux-amd64` (`gitea-runner v3.1.0`): all 23 tests, formatting, diff --git a/scripts/check-linux-container.sh b/scripts/check-linux-container.sh index a8b3544..13de280 100755 --- a/scripts/check-linux-container.sh +++ b/scripts/check-linux-container.sh @@ -13,6 +13,10 @@ check_root=$(mktemp -d /tmp/sized-check.XXXXXX) source_root=${SIZED_TEST_SOURCE:-/source} cp "$source_root/Cargo.toml" "$source_root/Cargo.lock" "$check_root/" cp -R "$source_root/src" "$source_root/tests" "$source_root/benches" "$check_root/" +cp "$source_root/LICENSE" "$source_root/README.md" "$source_root/MANUAL.md" "$check_root/" +mkdir "$check_root/scripts" +cp "$source_root/scripts/release.sh" "$source_root/scripts/check-release.sh" "$check_root/scripts/" +export SIZED_SOURCE_REVISION="$(git -C "$source_root" rev-parse HEAD 2>/dev/null || printf unknown)" cd "$check_root" cargo fmt --all -- --check cargo test --locked @@ -24,4 +28,5 @@ fixture_root=$(mktemp -d /tmp/sized-release.XXXXXX) printf 'Linux release smoke test\n' > "$fixture_root/payload" ./target/release/sized --version ./target/release/sized "$fixture_root" --threads 2 --one-file-system --apparent --format json +./scripts/check-release.sh printf 'Linux checks passed.\n' diff --git a/scripts/check-release.sh b/scripts/check-release.sh new file mode 100755 index 0000000..4fea86d --- /dev/null +++ b/scripts/check-release.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root=$(cd "$(dirname "$0")/.." && pwd) +check_root=$(mktemp -d "${TMPDIR:-/tmp}/sized-package.XXXXXX") +trap 'rm -rf "$check_root"' EXIT +"$repo_root/scripts/release.sh" --output-dir "$check_root/bundle" +mkdir "$check_root/extracted" "$check_root/fixture" +tar -xzf "$check_root/bundle.tar.gz" -C "$check_root/extracted" +for required in sized sized.1 sized.bash _sized sized.fish LICENSE README.md MANUAL.md BUILD-INFO.txt; do + test -s "$check_root/extracted/$required" +done +test -x "$check_root/extracted/sized" +( + cd "$check_root" + if command -v sha256sum >/dev/null 2>&1; then + sha256sum -c bundle.tar.gz.sha256 + else + shasum -a 256 -c bundle.tar.gz.sha256 + fi +) +printf 'owned package fixture\n' > "$check_root/fixture/payload" +"$check_root/extracted/sized" --version +"$check_root/extracted/sized" "$check_root/fixture" --apparent --format json > "$check_root/result.json" +grep -q 'payload' "$check_root/result.json" +grep -q '"complete":true' "$check_root/result.json" +if "$repo_root/scripts/release.sh" --output-dir "$check_root/bundle" > "$check_root/repeat.log" 2>&1; then + printf 'Packaging unexpectedly replaced an existing output.\n' >&2 + exit 1 +fi +grep -q 'Refusing to replace' "$check_root/repeat.log" +printf 'Archive contents, checksum, extracted CLI and overwrite guard passed.\n' diff --git a/scripts/release.sh b/scripts/release.sh index 8e2fe0d..d2c7b21 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -1,53 +1,56 @@ -#!/bin/bash -set -e +#!/usr/bin/env bash +set -euo pipefail -VERSION=$(grep '^version =' Cargo.toml | cut -d '"' -f 2) -DIST_DIR="dist/v$VERSION" - -echo "Building release assets for sized v$VERSION..." - -# 1. Clean and Prepare -rm -rf "$DIST_DIR" -mkdir -p "$DIST_DIR" - -# 2. Build Release Binary -cargo build --release - -# 3. Generate Man Page -cargo run --release -- --generate-man-page "$DIST_DIR" - -# 4. Generate Completions -cargo run --release -- --completions bash > "$DIST_DIR/sized.bash" -cargo run --release -- --completions zsh > "$DIST_DIR/_sized" -cargo run --release -- --completions fish > "$DIST_DIR/sized.fish" - -# 5. Build Debian Package (if cargo-deb is installed) -if command -v cargo-deb &> /dev/null; then - echo "Building Debian package..." - # On macOS, we use --no-strip to avoid 'unrecognized option: --strip-unneeded' - # and --no-build because we already built the release binary. - cargo deb --no-build --no-strip - cp target/debian/*.deb "$DIST_DIR/" - echo "Note: .deb package contains the binary for $(uname -s)-$(uname -m)" -else - echo "Warning: cargo-deb not found. Skipping .deb packaging." +# Build a host-target archive only. No tags, uploads, installers or signing. +repo_root=$(cd "$(dirname "$0")/.." && pwd) +cd "$repo_root" +version=$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml) +target=$(rustc -vV | sed -n 's/^host: //p') +revision=${SIZED_SOURCE_REVISION:-$(git rev-parse HEAD 2>/dev/null || printf unknown)} +destination="$repo_root/dist/sized-v$version-$target-${revision:0:12}" +if [[ $# == 2 && $1 == --output-dir ]]; then + destination=$2 +elif [[ $# != 0 ]]; then + printf 'Usage: %s [--output-dir ABSOLUTE_PATH]\n' "$0" >&2 + exit 2 fi +[[ $destination == /* ]] || { printf 'Output directory must be absolute.\n' >&2; exit 2; } +for output in "$destination" "$destination.tar.gz" "$destination.tar.gz.sha256"; do + [[ ! -e $output ]] || { printf 'Refusing to replace %s\n' "$output" >&2; exit 1; } +done -# 6. Build Alpine Package (Placeholder/Hook) -# Note: For real .apk building, one usually uses a docker container or abuild. -# This serves as a reminder for Alpine users. -if [ -f "APKBUILD" ] && command -v abuild &> /dev/null; then - echo "Building Alpine package..." - abuild -r -fi +target_dir=${CARGO_TARGET_DIR:-"$repo_root/target"} +[[ $target_dir == /* ]] || target_dir="$repo_root/$target_dir" +cargo build --locked --release --target "$target" +binary="$target_dir/$target/release/sized" +mkdir -p "$destination" +install -m 755 "$binary" "$destination/sized" +"$binary" --generate-man-page "$destination" +"$binary" --completions bash > "$destination/sized.bash" +"$binary" --completions zsh > "$destination/_sized" +"$binary" --completions fish > "$destination/sized.fish" +cp LICENSE README.md MANUAL.md "$destination/" +{ + printf 'Version: %s\nTarget: %s\nSource revision: %s\n' "$version" "$target" "$revision" + printf 'Source worktree: ' + if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + if git diff --quiet HEAD --; then printf 'clean tracked files\n'; else printf 'modified tracked files\n'; fi + else + printf 'exported source; verify against supplied revision\n' + fi + rustc --version + cargo --version + printf 'Unsigned host build; not a notarized Mac application.\n' +} > "$destination/BUILD-INFO.txt" -# 7. Create General Release Tarball -echo "Creating release tarball..." -tar -czf "dist/sized-v$VERSION-$(uname -s)-$(uname -m).tar.gz" -C "$DIST_DIR" . - -# 8. Copy Binary -cp target/release/sized "$DIST_DIR/" - -echo "Success! Assets are ready in $DIST_DIR" -ls -F "$DIST_DIR" -echo "Tarball: dist/sized-v$VERSION-$(uname -s)-$(uname -m).tar.gz" +tar -czf "$destination.tar.gz" -C "$destination" . +( + cd "$(dirname "$destination")" + archive="$(basename "$destination").tar.gz" + if command -v sha256sum >/dev/null 2>&1; then + sha256sum "$archive" > "$archive.sha256" + else + shasum -a 256 "$archive" > "$archive.sha256" + fi +) +printf 'Archive: %s.tar.gz\nChecksum: %s.tar.gz.sha256\n' "$destination" "$destination" -- 2.54.0 From 77b11a8c29b5ca3435fdd962717f8e8285abe972 Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Sat, 10 Oct 2026 04:00:52 -0400 Subject: [PATCH 8/9] Keep package targets explicit and reject existing archive symlinks --- CHANGELOG.md | 6 ++++++ scripts/check-release.sh | 7 +++++++ scripts/release.sh | 4 ++-- 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 74df2ea..c431669 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Fixed +- Include the executable before creating release archives. Use explicit host + targets, build provenance and checksums; refuse existing outputs and symlinks. +- Replace stale installation links and unavailable package-manager claims with + verified Gitea availability and explicit source-build instructions. - Count hard-link allocation once in a deterministic traversal order while retaining apparent sizes per pathname. Filtered comparison has independent allocation ownership, including when the first link is ignored. @@ -17,6 +21,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 symlinks as scan targets and distinguish special files from directories. ### Added +- Host archive extraction/executable checks on macOS and in Linux CI, plus + the Sized project page and its connection to SizeQueen's scanning core. - A scanner module with structured reports, file identities, progress counters, cooperative cancellation, and per-scan thread pools. - Optional `-x` / `--one-file-system` boundary handling. Partial CLI reports diff --git a/scripts/check-release.sh b/scripts/check-release.sh index 4fea86d..71b6fa2 100755 --- a/scripts/check-release.sh +++ b/scripts/check-release.sh @@ -29,4 +29,11 @@ if "$repo_root/scripts/release.sh" --output-dir "$check_root/bundle" > "$check_r exit 1 fi grep -q 'Refusing to replace' "$check_root/repeat.log" +ln -s "$check_root/untouched" "$check_root/linked.tar.gz" +if "$repo_root/scripts/release.sh" --output-dir "$check_root/linked" > "$check_root/linked.log" 2>&1; then + printf 'Packaging unexpectedly followed an existing archive symlink.\n' >&2 + exit 1 +fi +grep -q 'Refusing to replace' "$check_root/linked.log" +test ! -e "$check_root/untouched" printf 'Archive contents, checksum, extracted CLI and overwrite guard passed.\n' diff --git a/scripts/release.sh b/scripts/release.sh index d2c7b21..93c9f0a 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -16,12 +16,12 @@ elif [[ $# != 0 ]]; then fi [[ $destination == /* ]] || { printf 'Output directory must be absolute.\n' >&2; exit 2; } for output in "$destination" "$destination.tar.gz" "$destination.tar.gz.sha256"; do - [[ ! -e $output ]] || { printf 'Refusing to replace %s\n' "$output" >&2; exit 1; } + [[ ! -e $output && ! -L $output ]] || { printf 'Refusing to replace %s\n' "$output" >&2; exit 1; } done target_dir=${CARGO_TARGET_DIR:-"$repo_root/target"} [[ $target_dir == /* ]] || target_dir="$repo_root/$target_dir" -cargo build --locked --release --target "$target" +cargo build --locked --release --target "$target" --target-dir "$target_dir" binary="$target_dir/$target/release/sized" mkdir -p "$destination" install -m 755 "$binary" "$destination/sized" -- 2.54.0 From c2cc458f2788ed6ffb9781c1e2776872964d6bc6 Mon Sep 17 00:00:00 2001 From: Cole Speelman Date: Sat, 10 Oct 2026 04:09:31 -0400 Subject: [PATCH 9/9] Record live Sized project and verified release cleanup --- SHIPMENT.md | 10 +++++++--- TODO.md | 57 ++++++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 53 insertions(+), 14 deletions(-) diff --git a/SHIPMENT.md b/SHIPMENT.md index e24ea0e..1de4050 100644 --- a/SHIPMENT.md +++ b/SHIPMENT.md @@ -31,7 +31,8 @@ cliff-mads, overriding the job image with the pinned Sized Rust runtime. The runner itself launches the two tmpfs mounts. Jobs have no Docker socket and run `scripts/check-linux-container.sh` as UID 65532, using the checked-out workspace as `SIZED_TEST_SOURCE`. Its tests, strict Clippy, formatting and -optimized-binary smoke test are the same as the local Docker workflow. +optimized-binary smoke test and extracted release-archive verification are the +same as the local Docker workflow. The runtime contains Rust 1.88.0, rustfmt/Clippy and Node for the pinned checkout action. On cliff-mads, rebuild it explicitly with: @@ -55,8 +56,11 @@ The first complete native AMD64 push check is [Gitea run 1048 (number 2)](https://gitea.speelman.ca/gamertan/sized/actions/runs/1048), at `74b30bbf750417121f3b0c26017d2f011a2a8286`. All 23 tests, formatting, strict Clippy, release smoke and unchanged-checkout verification passed as UID 65532 -on `cliff-himesan-linux-amd64`. PR and manual-dispatch events are configured; -only push execution has been exercised. See `TODO.md` for the current checkpoint. +on `cliff-himesan-linux-amd64`. The packaging source checkpoint `77b11a8` +subsequently passed [push run 1065](https://gitea.speelman.ca/gamertan/sized/actions/runs/1065) +and [PR run 1066](https://gitea.speelman.ca/gamertan/sized/actions/runs/1066), +including executable/archive checks and output/symlink refusal. Manual dispatch +is configured but not independently exercised. See `TODO.md` for current work. ## 1. Versioning and Tagging diff --git a/TODO.md b/TODO.md index c09085c..25f99f1 100644 --- a/TODO.md +++ b/TODO.md @@ -18,7 +18,7 @@ Keep the existing CLI useful and preserve the GPL-3.0-only license. user, including actual mount boundaries, strict Clippy and a release smoke test. - [x] Commit the scanner hardening and Linux check tooling; push the existing `sizequeen-scan-hardening` review branch. Remote equality is verified. - Open a PR when ready; release/tag/package publication remains separate. + PR #2 is open; release/tag/package publication remains separate. - [x] Enable repository Actions and run the same Linux checks on the existing cliff-mads runner. Keep its container isolation and other jobs unchanged; verify a real workflow result before treating CI as proven. @@ -39,8 +39,9 @@ on SizeQueen. Website delivery is tracked in that repository's existing queue. Remove incidental installer generation; host binaries must not become `.deb`s. - [x] Verify archive contents/checksum, run the extracted CLI and test overwrite refusal on macOS arm64. Add the same verification to Linux CI. -- [ ] Verify the updated cliff-mads workflow, push the review checkpoint and - record the public CMS delivery. No new version, release assets or merge yet. +- [x] Verify the updated cliff-mads workflow, push the review checkpoint and + record the public CMS delivery. PR #2 is open; no new version, release assets + or merge. Historical v1.0.0 notes are corrected and asset bytes preserved. ## Deferred / next release @@ -54,8 +55,43 @@ scan tuning remain separate work. See `SHIPMENT.md` for the release process. ## Resume note -Current work: the approved project page and packaging cleanup above. Local Mac -archive verification passes; updated Linux CI and CMS publication are pending. +The approved Sized project page and release cleanup are delivered. Source +checkpoint `77b11a8c29b5ca3435fdd962717f8e8285abe972` is pushed on +`sizequeen-scan-hardening`; [PR #2](https://gitea.speelman.ca/gamertan/sized/pulls/2) +is open against unchanged `main`. Repository visibility remains public. + +Final cliff-mads checks passed for both events: +[push run 1065](https://gitea.speelman.ca/gamertan/sized/actions/runs/1065) +(4m 57s) and +[PR run 1066](https://gitea.speelman.ca/gamertan/sized/actions/runs/1066) +(5m 0s). Rust 1.88.0 / Linux AMD64 / UID 65532 passed all 23 tests, formatting, +strict Clippy, optimized smoke, archive extraction/checksum/executable and +existing-output/dangling-symlink guards, plus unchanged-checkout verification. +Mac arm64 passed the same package checks and documented Cargo/Make installs into +temporary prefixes. Ignored local evidence is in `target/release-audit/`, +including `gitea-run-1065.txt`, `package-macos.log` and `install-macos.log`. + +[Sized](https://gamertan.com/projects/sized/) is published through the shared +Gamertan project template (CMS revision 6), with a compact status sidebar, +verified synthetic CLI example, source instructions and accurate release limits. +It appears on the homepage, project index and sitemap. SizeQueen's +[Built on Sized section](https://gamertan.com/projects/sizequeen/) is published +in revision 14; all prior sections and app downloads are preserved. Desktop +and 320px mobile checks passed. Website evidence and recovery are recorded in +`../gamertancom-web-foundations/docs/PRODUCTION_SANDBOX_2026-09-04.md` under +“Sized project and Built on Sized — 10 October 2026”. + +Historical v1.0.0 release notes now identify the attached executable as macOS +arm64 and distinguish the newer review branch. Asset IDs, lengths and executable +SHA-256 above remain unchanged. No new binary release, tag, merge or private-core +source publication occurred. Next: review PR #2's CLI exit status, JSON and +library API changes, then choose a release version and complete corresponding +source/dependency notices before authorizing new binary publication. + +### Earlier verification history + +The records below describe previous checkpoints; current delivery and next +actions are above. Previous CI checkpoint: repository Actions is enabled and a real native AMD64 push run passed on cliff-mads. [Gitea run 1048 (number 2)](https://gitea.speelman.ca/gamertan/sized/actions/runs/1048) @@ -69,7 +105,8 @@ the corrected runtime pre-owns it for the job user. The workflow reuses the existing `himesan-node24` label and a local, pinned Rust/Node image. Runner configuration, other jobs and repository visibility are unchanged. Temporary setup credentials were revoked and their files removed. Push execution is -proven; PR/manual-dispatch triggers are configured but not independently run. +proven; PR execution was subsequently proven in run 1066; manual dispatch remains +configured but not independently exercised. Linux validation and the requested remote review branch are complete. Scanner hardening is `8b177e2`; repeatable Linux checks and equivalent format @@ -79,7 +116,7 @@ unprivileged checkout fix `74b30bb`. The source checkpoint is verified remotely at `74b30bbf750417121f3b0c26017d2f011a2a8286`; later documentation-only checkpoint commits do not rerun the source checks. Gitea main remains `9c8d1d4`. The repository was already public (`private: false`); visibility is unchanged. -No PR, merge, tag, package publication or release occurred. +At that checkpoint, no PR, merge, tag, package publication or release occurred. `scripts/check-linux.sh` owns the container workflow. Linux ARM64 (native in Docker's VM) and AMD64 (emulated on this Mac) each passed 23 tests as UID 65532: @@ -96,10 +133,8 @@ Rust 1.88 Clippy identified format-string style warnings; equivalent interpolations fix those without suppressions. SizeQueen's included scanner still matches `8b177e2` exactly; the follow-up changes only one scanner Display format string, not scan behaviour. SizeQueen itself was unchanged in this pass. -Next: open/review a PR against main, calling out CLI partial-scan status and -library/API changes. Keep shared-core extraction as the following proposed PR, -then choose a release version and repair/verify packaging when release work -is authorized. These checks do not prove desktop X11/Wayland interaction, +The planned PR and packaging repairs are now complete; review and version +selection remain next. These checks do not prove desktop X11/Wayland interaction, Windows allocation or performance on very large/cold/remote trees. Previous local checkpoint: baseline `9c8d1d4` matched Gitea main; -- 2.54.0