Morph MORPH
SIGN IN SIGN UP
ziglang / zig UNCLAIMED

Moved to Codeberg

42798 0 0 Zig
Normal View History Raw
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
//! Manages `zig-cache` directories.
//! This is not a general-purpose cache. It is designed to be fast and simple,
//! not to withstand attacks using specially-crafted input.
allocgate: std Allocator interface refactor
2021-10-29 00:37:25 +01:00
gpa: Allocator,
move std.cache_hash from std to stage2 The API is pretty specific to the implementationt details of the self-hosted compiler. I don't want to have to independently support and maintain this as part of the standard library, and be obligated to not make breaking changes to it with changes to the implementation of stage2.
2020-09-14 11:05:51 -07:00
manifest_dir: fs.Dir,
hash: HashHelper = .{},
Cache: use mutex to protect recent_problematic_timestamp The previous commit tried to use atomics but not many CPUs support 128-bit atomics. So we use a mutex. In order to avoid contention, we also store `recent_problematic_timestamp` locally on the `Manifest` which is only ever accessed from a single thread at a time, and only consult the global one if the local one is problematic. This commit was tested by running `zig build test-behavior` in two separate terminals at the same time.
2021-12-09 22:07:28 -07:00
/// This value is accessed from multiple threads, protected by mutex.
Cache: improvements to previous commit * put `recent_problematic_timestamp` onto `Cache` so that it can be shared by multiple Manifest instances. * make `isProblematicTimestamp` return true on any filesystem error. * save 1 syscall by using truncate=true in createFile instead of calling `setEndPos`.
2021-12-09 18:55:20 -07:00
recent_problematic_timestamp: i128 = 0,
Cache: use mutex to protect recent_problematic_timestamp The previous commit tried to use atomics but not many CPUs support 128-bit atomics. So we use a mutex. In order to avoid contention, we also store `recent_problematic_timestamp` locally on the `Manifest` which is only ever accessed from a single thread at a time, and only consult the global one if the local one is problematic. This commit was tested by running `zig build test-behavior` in two separate terminals at the same time.
2021-12-09 22:07:28 -07:00
mutex: std.Thread.Mutex = .{},
move std.cache_hash from std to stage2 The API is pretty specific to the implementationt details of the self-hosted compiler. I don't want to have to independently support and maintain this as part of the standard library, and be obligated to not make breaking changes to it with changes to the implementation of stage2.
2020-09-14 11:05:51 -07:00
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
/// A set of strings such as the zig library directory or project source root, which
/// are stripped from the file paths before putting into the cache. They
/// are replaced with single-character indicators. This is not to save
/// space but to eliminate absolute file paths. This improves portability
/// and usefulness of the cache for advanced use cases.
std.Build.Cache: remove debug log statements Now that this API is used by the build system, these debug logs are problematic because build scripts run in debug mode, making these logs noisy output.
2023-02-09 10:00:25 -07:00
prefixes_buffer: [4]Directory = undefined,
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
prefixes_len: usize = 0,
move Package.Path to std.Build.Cache.Path
2024-03-21 16:16:47 -07:00
pub const Path = @import("Cache/Path.zig");
extract std.Build.Cache.Directory into separate file
2024-03-21 16:11:59 -07:00
pub const Directory = @import("Cache/Directory.zig");
move the cache system from compiler to std lib
2023-02-05 19:39:04 -07:00
pub const DepTokenizer = @import("Cache/DepTokenizer.zig");
move std.cache_hash from std to stage2 The API is pretty specific to the implementationt details of the self-hosted compiler. I don't want to have to independently support and maintain this as part of the standard library, and be obligated to not make breaking changes to it with changes to the implementation of stage2.
2020-09-14 11:05:51 -07:00
const Cache = @This();
const std = @import("std");
migrate from `std.Target.current` to `@import("builtin").target` closes #9388 closes #9321
2021-10-04 23:47:27 -07:00
const builtin = @import("builtin");
cache_hash: hash function change This makes the `cache_hash` hash function easier to replace. BLAKE3 would be a natural fit for hashing large files, but: - second preimage resistance is not necessary for the cache_hash use cases - our BLAKE3 implementation is currently very slow Switch to SipHash128, which gives us an immediate speed boost.
2020-08-21 15:08:15 +02:00
const crypto = std.crypto;
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
const fs = std.fs;
const assert = std.debug.assert;
const testing = std.testing;
const mem = std.mem;
const fmt = std.fmt;
const Allocator = std.mem.Allocator;
Cache: add debug log statement
2021-11-24 23:08:37 -07:00
const log = std.log.scoped(.cache);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
move the cache system from compiler to std lib
2023-02-05 19:39:04 -07:00
pub fn addPrefix(cache: *Cache, directory: Directory) void {
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
cache.prefixes_buffer[cache.prefixes_len] = directory;
cache.prefixes_len += 1;
}
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
/// Be sure to call `Manifest.deinit` after successful initialization.
Cache: improvements to previous commit * put `recent_problematic_timestamp` onto `Cache` so that it can be shared by multiple Manifest instances. * make `isProblematicTimestamp` return true on any filesystem error. * save 1 syscall by using truncate=true in createFile instead of calling `setEndPos`.
2021-12-09 18:55:20 -07:00
pub fn obtain(cache: *Cache) Manifest {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
return .{
move std.cache_hash from std to stage2 The API is pretty specific to the implementationt details of the self-hosted compiler. I don't want to have to independently support and maintain this as part of the standard library, and be obligated to not make breaking changes to it with changes to the implementation of stage2.
2020-09-14 11:05:51 -07:00
.cache = cache,
.hash = cache.hash,
.manifest_file = null,
.manifest_dirty = false,
stage2 Cache: use hex instead of base64 for file paths
2020-09-16 12:31:42 -07:00
.hex_digest = undefined,
move std.cache_hash from std to stage2 The API is pretty specific to the implementationt details of the self-hosted compiler. I don't want to have to independently support and maintain this as part of the standard library, and be obligated to not make breaking changes to it with changes to the implementation of stage2.
2020-09-14 11:05:51 -07:00
};
}
move the cache system from compiler to std lib
2023-02-05 19:39:04 -07:00
pub fn prefixes(cache: *const Cache) []const Directory {
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
return cache.prefixes_buffer[0..cache.prefixes_len];
}
const PrefixedPath = struct {
prefix: u8,
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
sub_path: []const u8,
fn eql(a: PrefixedPath, b: PrefixedPath) bool {
return a.prefix == b.prefix and std.mem.eql(u8, a.sub_path, b.sub_path);
}
fn hash(pp: PrefixedPath) u32 {
return @truncate(std.hash.Wyhash.hash(pp.prefix, pp.sub_path));
}
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
};
fn findPrefix(cache: *const Cache, file_path: []const u8) !PrefixedPath {
const gpa = cache.gpa;
const resolved_path = try fs.path.resolve(gpa, &[_][]const u8{file_path});
errdefer gpa.free(resolved_path);
return findPrefixResolved(cache, resolved_path);
}
/// Takes ownership of `resolved_path` on success.
fn findPrefixResolved(cache: *const Cache, resolved_path: []u8) !PrefixedPath {
const gpa = cache.gpa;
const prefixes_slice = cache.prefixes();
var i: u8 = 1; // Start at 1 to skip over checking the null prefix.
while (i < prefixes_slice.len) : (i += 1) {
const p = prefixes_slice[i].path.?;
lib: correct unnecessary uses of 'var'
2023-11-10 05:27:17 +00:00
const sub_path = getPrefixSubpath(gpa, p, resolved_path) catch |err| switch (err) {
Cache: Fix findPrefix when paths are slightly out of the ordinary This makes Cache.findPrefix/findPrefixResolved use `std.fs.path.relative` instead of `std.mem.startsWith` when checking if a file is within a prefix. This fixes multiple edge cases around prefix detection: - If a prefix path ended with a path separator, then the first character of the 'sub_path' would get cut off because the previous implementation assumed it was a path separator. Example: prefix: `/foo/`, file_path: `/foo/abc.txt` would see that they both start with `/foo/` and then slice starting from one byte past the common prefix, ending up with `bc.txt` instead of the expected `abc.txt` - If a prefix contained double path separators after any component, then the `startsWith` check would erroneously fail. Example: prefix: `/foo//bar`, file_path: `/foo/bar/abc.txt` would not see that abc.txt is a sub path of the prefix `/foo//bar` - On Windows, case insensitivity was not respected at all, instead the UTF-8 bytes were compared directly This fixes all of the things in the above list (and possibly more).
2023-08-19 15:41:09 -07:00
error.NotASubPath => continue,
else => |e| return e,
};
// Free the resolved path since we're not going to return it
gpa.free(resolved_path);
return PrefixedPath{
.prefix = i,
.sub_path = sub_path,
};
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
}
return PrefixedPath{
.prefix = 0,
.sub_path = resolved_path,
};
}
Cache: Fix findPrefix when paths are slightly out of the ordinary This makes Cache.findPrefix/findPrefixResolved use `std.fs.path.relative` instead of `std.mem.startsWith` when checking if a file is within a prefix. This fixes multiple edge cases around prefix detection: - If a prefix path ended with a path separator, then the first character of the 'sub_path' would get cut off because the previous implementation assumed it was a path separator. Example: prefix: `/foo/`, file_path: `/foo/abc.txt` would see that they both start with `/foo/` and then slice starting from one byte past the common prefix, ending up with `bc.txt` instead of the expected `abc.txt` - If a prefix contained double path separators after any component, then the `startsWith` check would erroneously fail. Example: prefix: `/foo//bar`, file_path: `/foo/bar/abc.txt` would not see that abc.txt is a sub path of the prefix `/foo//bar` - On Windows, case insensitivity was not respected at all, instead the UTF-8 bytes were compared directly This fixes all of the things in the above list (and possibly more).
2023-08-19 15:41:09 -07:00
fn getPrefixSubpath(allocator: Allocator, prefix: []const u8, path: []u8) ![]u8 {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
const relative = try fs.path.relative(allocator, prefix, path);
Cache: Fix findPrefix when paths are slightly out of the ordinary This makes Cache.findPrefix/findPrefixResolved use `std.fs.path.relative` instead of `std.mem.startsWith` when checking if a file is within a prefix. This fixes multiple edge cases around prefix detection: - If a prefix path ended with a path separator, then the first character of the 'sub_path' would get cut off because the previous implementation assumed it was a path separator. Example: prefix: `/foo/`, file_path: `/foo/abc.txt` would see that they both start with `/foo/` and then slice starting from one byte past the common prefix, ending up with `bc.txt` instead of the expected `abc.txt` - If a prefix contained double path separators after any component, then the `startsWith` check would erroneously fail. Example: prefix: `/foo//bar`, file_path: `/foo/bar/abc.txt` would not see that abc.txt is a sub path of the prefix `/foo//bar` - On Windows, case insensitivity was not respected at all, instead the UTF-8 bytes were compared directly This fixes all of the things in the above list (and possibly more).
2023-08-19 15:41:09 -07:00
errdefer allocator.free(relative);
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
var component_iterator = fs.path.NativeComponentIterator.init(relative) catch {
Cache: Fix findPrefix when paths are slightly out of the ordinary This makes Cache.findPrefix/findPrefixResolved use `std.fs.path.relative` instead of `std.mem.startsWith` when checking if a file is within a prefix. This fixes multiple edge cases around prefix detection: - If a prefix path ended with a path separator, then the first character of the 'sub_path' would get cut off because the previous implementation assumed it was a path separator. Example: prefix: `/foo/`, file_path: `/foo/abc.txt` would see that they both start with `/foo/` and then slice starting from one byte past the common prefix, ending up with `bc.txt` instead of the expected `abc.txt` - If a prefix contained double path separators after any component, then the `startsWith` check would erroneously fail. Example: prefix: `/foo//bar`, file_path: `/foo/bar/abc.txt` would not see that abc.txt is a sub path of the prefix `/foo//bar` - On Windows, case insensitivity was not respected at all, instead the UTF-8 bytes were compared directly This fixes all of the things in the above list (and possibly more).
2023-08-19 15:41:09 -07:00
return error.NotASubPath;
};
if (component_iterator.root() != null) {
return error.NotASubPath;
}
const first_component = component_iterator.first();
if (first_component != null and std.mem.eql(u8, first_component.?.name, "..")) {
return error.NotASubPath;
}
return relative;
}
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
/// This is 128 bits - Even with 2^54 cache entries, the probably of a collision would be under 10^-6
stage2 Cache: use hex instead of base64 for file paths
2020-09-16 12:31:42 -07:00
pub const bin_digest_len = 16;
pub const hex_digest_len = bin_digest_len * 2;
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
pub const BinDigest = [bin_digest_len]u8;
std.Build.Cache: add HexDigest type
2023-12-11 23:08:03 +01:00
pub const HexDigest = [hex_digest_len]u8;
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
/// This is currently just an arbitrary non-empty string that can't match another manifest line.
const manifest_header = "0";
std.Build.Cache: bump manifest_file_size_max to 100M Some users are hitting this limit. I think it's primarily due to not deduplicating (solved in the previous commit) but this seems like a better limit regardless.
2024-03-21 19:56:47 -07:00
const manifest_file_size_max = 100 * 1024 * 1024;
Set manifest's maximum size to Andrew's recommendation
2020-04-30 20:00:26 -06:00
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
/// The type used for hashing file contents. Currently, this is SipHash128(1, 3), because it
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
/// provides enough collision resistance for the Manifest use cases, while being one of our
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
/// fastest options right now.
pub const Hasher = crypto.auth.siphash.SipHash128(1, 3);
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
/// Initial state with random bytes, that can be copied.
/// Refresh this with new random bytes when the manifest
/// format is modified in a non-backwards-compatible way.
pub const hasher_init: Hasher = Hasher.init(&[_]u8{
0x33, 0x52, 0xa2, 0x84,
0xcf, 0x17, 0x56, 0x57,
0x01, 0xbb, 0xcd, 0xe4,
0x77, 0xd6, 0xf0, 0x60,
});
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
Rename CacheHashFile -> File
2020-03-06 20:23:15 -07:00
pub const File = struct {
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
prefixed_path: PrefixedPath,
Add max_file_size argument
2020-05-01 23:06:10 -06:00
max_file_size: ?usize,
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
/// Populated if the user calls `addOpenedFile`.
/// The handle is not owned here.
handle: ?fs.File,
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
stat: Stat,
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
bin_digest: BinDigest,
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
contents: ?[]const u8,
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
pub const Stat = struct {
inode: fs.File.INode,
size: u64,
mtime: i128,
link.Elf: eliminate an O(N^2) algorithm in flush() Make shared_objects a StringArrayHashMap so that deduping does not need to happen in flush. That deduping code also was using an O(N^2) algorithm, which is not allowed in this codebase. There is another violation of this rule in resolveSymbols but this commit does not address it. This required reworking shared object parsing, breaking it into independent components so that we could access soname earlier. Shared object parsing had a few problems that I noticed and fixed in this commit: * Many instances of incorrect use of align(1). * `shnum * @sizeOf(elf.Elf64_Shdr)` can overflow based on user data. * `@divExact` can cause illegal behavior based on user data. * Strange versyms logic that wasn't present in mold nor lld. The logic was not commented and there is no git blame information in ziglang/zig nor kubkon/zld. I changed it to match mold and lld instead. * Use of ArrayList for slices of memory that are never resized. * finding DT_VERDEFNUM in a different loop than finding DT_SONAME. Ultimately I think we should follow mold's lead and ignore this integer, relying on null termination instead. * Doing logic based on VER_FLG_BASE rather than ignoring it like mold and LLD do. No comment explaining why the behavior is different. * Mutating the original ELF symbols rather than only storing the mangled name on the new Symbol struct. I noticed something that I didn't try to address in this commit: Symbol stores a lot of redundant information that is already present in the ELF symbols. I suspect that the codebase could benefit from reworking Symbol to not store redundant information. Additionally: * Add some type safety to std.elf. * Eliminate 1-3 file system reads for determining the kind of input files, by taking advantage of file name extension and handling error codes properly. * Move more error handling methods to link.Diags and make them infallible and thread-safe * Make the data dependencies obvious in the parameters of parseSharedObject. It's now clear that the first two steps (Header and Parsed) can be done during the main Compilation pipeline, rather than waiting for flush().
2024-10-11 23:28:31 -07:00
pub fn fromFs(fs_stat: fs.File.Stat) Stat {
return .{
.inode = fs_stat.inode,
.size = fs_stat.size,
.mtime = fs_stat.mtime,
};
}
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
};
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
pub fn deinit(self: *File, gpa: Allocator) void {
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
gpa.free(self.prefixed_path.sub_path);
Return an index from `CacheHash.addFile` This makes it possible for the user to retrieve the contents of the file without running into data races.
2020-04-15 20:13:26 -06:00
if (self.contents) |contents| {
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
gpa.free(contents);
Return an index from `CacheHash.addFile` This makes it possible for the user to retrieve the contents of the file without running into data races.
2020-04-15 20:13:26 -06:00
self.contents = null;
}
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
self.* = undefined;
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
pub fn updateMaxSize(file: *File, new_max_size: ?usize) void {
const new = new_max_size orelse return;
file.max_file_size = if (file.max_file_size) |old| @max(old, new) else new;
}
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
pub fn updateHandle(file: *File, new_handle: ?fs.File) void {
const handle = new_handle orelse return;
file.handle = handle;
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
};
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
pub const HashHelper = struct {
hasher: Hasher = hasher_init,
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Improvements to docs and text * docs(std.math): elaborate on difference between absCast and absInt * docs(std.rand.Random.weightedIndex): elaborate on likelihood I think this makes it easier to understand. * langref: add small reminder * docs(std.fs.path.extension): brevity * docs(std.bit_set.StaticBitSet): mention the specific types * std.debug.TTY: explain what purpose this struct serves This should also make it clearer that this struct is not supposed to provide unrelated terminal manipulation functionality such as setting the cursor position or something because terminals are complicated and we should keep this struct simple and focused on debugging. * langref(package listing): brevity * langref: explain what exactly `threadlocal` causes to happen * std.array_list: link between swapRemove and orderedRemove Maybe this can serve as a TLDR and make it easier to decide. * PrefetchOptions.locality: clarify docs that this is a range This confused me previously and I thought I can only use either 0 or 3. * fix typos and more * std.builtin.CallingConvention: document some CCs * langref: explain possibly cryptic names I think it helps knowing what exactly these acronyms (@clz and @ctz) and abbreviations (@popCount) mean. * variadic function error: add missing preposition * std.fmt.format docs: nicely hyphenate * help menu: say what to optimize for I think this is slightly more specific than just calling it "optimizations". These are speed optimizations. I used the word "performance" here.
2023-04-23 20:06:21 +02:00
/// Record a slice of bytes as a dependency of the process being cached.
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
pub fn addBytes(hh: *HashHelper, bytes: []const u8) void {
hh.hasher.update(mem.asBytes(&bytes.len));
hh.hasher.update(bytes);
Add `cache` method; add support for caching integers
2020-03-05 21:32:26 -07:00
}
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
pub fn addOptionalBytes(hh: *HashHelper, optional_bytes: ?[]const u8) void {
hh.add(optional_bytes != null);
hh.addBytes(optional_bytes orelse return);
}
Make type specific add functions Basically, move type specific code into their own functions instead of making `add` a giant function responsible for everything.
2020-03-06 21:40:33 -07:00
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
pub fn addListOfBytes(hh: *HashHelper, list_of_bytes: []const []const u8) void {
stage2: caching system integration & Module/Compilation splitting * update to the new cache hash API * std.Target defaultVersionRange moves to std.Target.Os.Tag * std.Target.Os gains getVersionRange which returns a tagged union * start the process of splitting Module into Compilation and "zig module". - The parts of Module having to do with only compiling zig code are extracted into ZigModule.zig. - Next step is to rename Module to Compilation. - After that rename ZigModule back to Module. * implement proper cache hash usage when compiling C objects, and properly manage the file lock of the build artifacts. * make versions optional to match recent changes to master branch. * proper cache hash integration for compiling zig code * proper cache hash integration for linking even when not compiling zig code. * ELF LLD linking integrates with the caching system. A comment from the source code: Here we want to determine whether we can save time by not invoking LLD when the output is unchanged. None of the linker options or the object files that are being linked are in the hash that namespaces the directory we are outputting to. Therefore, we must hash those now, and the resulting digest will form the "id" of the linking job we are about to perform. After a successful link, we store the id in the metadata of a symlink named "id.txt" in the artifact directory. So, now, we check if this symlink exists, and if it matches our digest. If so, we can skip linking. Otherwise, we proceed with invoking LLD. * implement disable_c_depfile option * add tracy to a few more functions
2020-09-13 19:17:58 -07:00
hh.add(list_of_bytes.len);
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
for (list_of_bytes) |bytes| hh.addBytes(bytes);
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
}
objcopy: support multiple only sections
2024-03-01 09:18:33 +08:00
pub fn addOptionalListOfBytes(hh: *HashHelper, optional_list_of_bytes: ?[]const []const u8) void {
hh.add(optional_list_of_bytes != null);
hh.addListOfBytes(optional_list_of_bytes orelse return);
}
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
/// Convert the input value into bytes and record it as a dependency of the process being cached.
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
pub fn add(hh: *HashHelper, x: anytype) void {
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
switch (@TypeOf(x)) {
std: replace builtin.Version with SemanticVersion
2023-02-21 18:39:22 +01:00
std.SemanticVersion => {
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
hh.add(x.major);
hh.add(x.minor);
hh.add(x.patch);
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
},
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
std.Target.Os.TaggedVersionRange => {
switch (x) {
std.Target: Add Os.HurdVersionRange for Os.Tag.hurd. This is necessary since isGnuLibC() is true for hurd, so we need to be able to represent a glibc version for it. Also add an Os.TaggedVersionRange.gnuLibCVersion() convenience function.
2024-11-23 17:57:39 +01:00
.hurd => |hurd| {
hh.add(hurd.range.min);
hh.add(hurd.range.max);
hh.add(hurd.glibc);
},
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
.linux => |linux| {
hh.add(linux.range.min);
hh.add(linux.range.max);
hh.add(linux.glibc);
std.Target: Add support for specifying Android API level.
2024-10-30 21:57:44 +01:00
hh.add(linux.android);
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
},
.windows => |windows| {
hh.add(windows.min);
hh.add(windows.max);
},
.semver => |semver| {
hh.add(semver.min);
hh.add(semver.max);
},
.none => {},
}
},
zig build system: change target, compilation, and module APIs Introduce the concept of "target query" and "resolved target". A target query is what the user specifies, with some things left to default. A resolved target has the default things discovered and populated. In the future, std.zig.CrossTarget will be rename to std.Target.Query. Introduces `std.Build.resolveTargetQuery` to get from one to the other. The concept of `main_mod_path` is gone, no longer supported. You have to put the root source file at the module root now. * remove deprecated API * update build.zig for the breaking API changes in this branch * move std.Build.Step.Compile.BuildId to std.zig.BuildId * add more options to std.Build.ExecutableOptions, std.Build.ObjectOptions, std.Build.SharedLibraryOptions, std.Build.StaticLibraryOptions, and std.Build.TestOptions. * remove `std.Build.constructCMacro`. There is no use for this API. * deprecate `std.Build.Step.Compile.defineCMacro`. Instead, `std.Build.Module.addCMacro` is provided. - remove `std.Build.Step.Compile.defineCMacroRaw`. * deprecate `std.Build.Step.Compile.linkFrameworkNeeded` - use `std.Build.Module.linkFramework` * deprecate `std.Build.Step.Compile.linkFrameworkWeak` - use `std.Build.Module.linkFramework` * move more logic into `std.Build.Module` * allow `target` and `optimize` to be `null` when creating a Module. Along with other fields, those unspecified options will be inherited from parent `Module` when inserted into an import table. * the `target` field of `addExecutable` is now required. pass `b.host` to get the host target.
2023-12-02 21:51:34 -07:00
std.zig.BuildId => switch (x) {
tweaks to --build-id * build.zig: the result of b.option() can be assigned directly in many cases thanks to the return type being an optional * std.Build: make the build system aware of the std.Build.Step.Compile.BuildId type when used as an option. - remove extraneous newlines in error logs * simplify caching logic * simplify hexstring parsing tests and use a doc test * simplify hashing logic. don't use an optional when the `none` tag already provides this meaning. * CLI: fix incorrect linker arg parsing
2023-05-16 20:00:47 -07:00
.none, .fast, .uuid, .sha1, .md5 => hh.add(std.meta.activeTag(x)),
.hexstring => |hex_string| hh.addBytes(hex_string.toSlice()),
},
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
else => switch (@typeInfo(@TypeOf(x))) {
std: update `std.builtin.Type` fields to follow naming conventions The compiler actually doesn't need any functional changes for this: Sema does reification based on the tag indices of `std.builtin.Type` already! So, no zig1.wasm update is necessary. This change is necessary to disallow name clashes between fields and decls on a type, which is a prerequisite of #9938.
2024-08-28 02:35:53 +01:00
.bool, .int, .@"enum", .array => hh.addBytes(mem.asBytes(&x)),
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
else => @compileError("unable to hash type " ++ @typeName(@TypeOf(x))),
},
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
pub fn addOptional(hh: *HashHelper, optional: anytype) void {
hh.add(optional != null);
hh.add(optional orelse return);
}
stage2 Cache: use hex instead of base64 for file paths
2020-09-16 12:31:42 -07:00
/// Returns a hex encoded hash of the inputs, without modifying state.
pub fn peek(hh: HashHelper) [hex_digest_len]u8 {
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
var copy = hh;
return copy.final();
}
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
pub fn peekBin(hh: HashHelper) BinDigest {
stage2: implement zig build As part of this: * add std.process.cleanExit. closes #6395 - use it in several places * adjust the alignment of text in `zig build --help` menu * Cache: support the concept of "unhit" so that we properly keep track of the cache when we find out using the secondary hash that the cache "hit" was actually a miss. Use this to fix false negatives of caching of stage1 build artifacts. * fix not deleting the symlink hash for stage1 build artifacts causing false positives. * implement support for Package arguments in stage1 build artifacts * update and add missing usage text * add --override-lib-dir and --enable-cache CLI options - `--enable-cache` takes the place of `--cache on` * CLI supports -femit-bin=foo combined with --enable-cache to do an "update file" operation. --enable-cache without that argument will build the output into a cache directory and then print the path to stdout (matching master branch behavior). * errors surfacing from main() now print "error: Foo" instead of "error: error.Foo".
2020-09-22 22:18:19 -07:00
var copy = hh;
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
var bin_digest: BinDigest = undefined;
stage2: implement zig build As part of this: * add std.process.cleanExit. closes #6395 - use it in several places * adjust the alignment of text in `zig build --help` menu * Cache: support the concept of "unhit" so that we properly keep track of the cache when we find out using the secondary hash that the cache "hit" was actually a miss. Use this to fix false negatives of caching of stage1 build artifacts. * fix not deleting the symlink hash for stage1 build artifacts causing false positives. * implement support for Package arguments in stage1 build artifacts * update and add missing usage text * add --override-lib-dir and --enable-cache CLI options - `--enable-cache` takes the place of `--cache on` * CLI supports -femit-bin=foo combined with --enable-cache to do an "update file" operation. --enable-cache without that argument will build the output into a cache directory and then print the path to stdout (matching master branch behavior). * errors surfacing from main() now print "error: Foo" instead of "error: error.Foo".
2020-09-22 22:18:19 -07:00
copy.hasher.final(&bin_digest);
return bin_digest;
}
stage2 Cache: use hex instead of base64 for file paths
2020-09-16 12:31:42 -07:00
/// Returns a hex encoded hash of the inputs, mutating the state of the hasher.
std.Build.Cache: add HexDigest type
2023-12-11 23:08:03 +01:00
pub fn final(hh: *HashHelper) HexDigest {
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
var bin_digest: BinDigest = undefined;
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
hh.hasher.final(&bin_digest);
std.Build.Cache: add binToHex function reduces need for API users to rely on formatted printing, even though that's how it is currently implemented.
2024-07-04 14:15:15 -07:00
return binToHex(bin_digest);
WIP: move many global settings to become per-Module Much of the logic from Compilation.create() is extracted into Compilation.Config.resolve() which accepts many optional settings and produces concrete settings. This separate step is needed by API users of Compilation so that they can pass the resolved global settings to the Module creation function, which itself needs to resolve per-Module settings. Since the target and other things are no longer global settings, I did not want them stored in link.File (in the `options` field). That options field was already a kludge; those options should be resolved into concrete settings. This commit also starts to work on that, deleting link.Options, moving the fields into Compilation and ObjectFormat-specific structs instead. Some fields were ephemeral and should not have been stored at all, such as symbol_size_hint. The link.File object of Compilation is now a `?*link.File` and `null` when -fno-emit-bin is passed. It is now arena-allocated along with Compilation itself, avoiding some messy cleanup code that was there before. On the command line, it is now possible to configure the standard library itself by using `--mod std` just like any other module. This meant that the CLI needed to create the standard library module rather than having Compilation create it. There are a lot of changes in this commit and it's still not done. I didn't realize how quickly this changeset was going to balloon out of control, and there are still many lines that need to be changed before it even compiles successfully. * introduce std.Build.Cache.HashHelper.oneShot * add error_tracing to std.Build.Module * extract build.zig file generation into src/Builtin.zig * each CSourceFile and RcSourceFile now has a Module owner, which determines some of the C compiler flags.
2023-12-10 15:25:06 -07:00
}
pub fn oneShot(bytes: []const u8) [hex_digest_len]u8 {
var hasher: Hasher = hasher_init;
hasher.update(bytes);
var bin_digest: BinDigest = undefined;
hasher.final(&bin_digest);
std.Build.Cache: add binToHex function reduces need for API users to rely on formatted printing, even though that's how it is currently implemented.
2024-07-04 14:15:15 -07:00
return binToHex(bin_digest);
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
}
};
std.Build.Cache: add binToHex function reduces need for API users to rely on formatted printing, even though that's how it is currently implemented.
2024-07-04 14:15:15 -07:00
pub fn binToHex(bin_digest: BinDigest) HexDigest {
var out_digest: HexDigest = undefined;
_ = fmt.bufPrint(
&out_digest,
"{s}",
.{fmt.fmtSliceHexLower(&bin_digest)},
) catch unreachable;
return out_digest;
}
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
pub const Lock = struct {
manifest_file: fs.File,
pub fn release(lock: *Lock) void {
cache: Fix LockViolation during C compilation paths (#13591) - C compilation flows didn't hold an exclusive lock on the cache manifest file when writing to it in all cases - On windows, explicitly unlock the file lock before closing it
2022-12-06 23:15:54 -05:00
if (builtin.os.tag == .windows) {
// Windows does not guarantee that locks are immediately unlocked when
// the file handle is closed. See LockFileEx documentation.
lock.manifest_file.unlock();
}
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
lock.manifest_file.close();
lock.* = undefined;
}
};
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub const Manifest = struct {
Cache: improvements to previous commit * put `recent_problematic_timestamp` onto `Cache` so that it can be shared by multiple Manifest instances. * make `isProblematicTimestamp` return true on any filesystem error. * save 1 syscall by using truncate=true in createFile instead of calling `setEndPos`.
2021-12-09 18:55:20 -07:00
cache: *Cache,
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// Current state for incremental hashing.
hash: HashHelper,
manifest_file: ?fs.File,
manifest_dirty: bool,
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
/// Set this flag to true before calling hit() in order to indicate that
/// upon a cache hit, the code using the cache will not modify the files
/// within the cache directory. This allows multiple processes to utilize
/// the same cache directory at the same time.
want_shared_lock: bool = true,
have_exclusive_lock: bool = false,
Cache: use mutex to protect recent_problematic_timestamp The previous commit tried to use atomics but not many CPUs support 128-bit atomics. So we use a mutex. In order to avoid contention, we also store `recent_problematic_timestamp` locally on the `Manifest` which is only ever accessed from a single thread at a time, and only consult the global one if the local one is problematic. This commit was tested by running `zig build test-behavior` in two separate terminals at the same time.
2021-12-09 22:07:28 -07:00
// Indicate that we want isProblematicTimestamp to perform a filesystem write in
// order to obtain a problematic timestamp for the next call. Calls after that
// will then use the same timestamp, to avoid unnecessary filesystem writes.
want_refresh_timestamp: bool = true,
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
files: Files = .{},
std.Build.Cache: add HexDigest type
2023-12-11 23:08:03 +01:00
hex_digest: HexDigest,
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
diagnostic: Diagnostic = .none,
Cache: use mutex to protect recent_problematic_timestamp The previous commit tried to use atomics but not many CPUs support 128-bit atomics. So we use a mutex. In order to avoid contention, we also store `recent_problematic_timestamp` locally on the `Manifest` which is only ever accessed from a single thread at a time, and only consult the global one if the local one is problematic. This commit was tested by running `zig build test-behavior` in two separate terminals at the same time.
2021-12-09 22:07:28 -07:00
/// Keeps track of the last time we performed a file system write to observe
/// what time the file system thinks it is, according to its own granularity.
recent_problematic_timestamp: i128 = 0,
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
pub const Diagnostic = union(enum) {
none,
manifest_create: fs.File.OpenError,
manifest_read: fs.File.ReadError,
manifest_lock: fs.File.LockError,
file_open: FileOp,
file_stat: FileOp,
file_read: FileOp,
file_hash: FileOp,
pub const FileOp = struct {
file_index: usize,
err: anyerror,
};
};
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
pub const Files = std.ArrayHashMapUnmanaged(File, void, FilesContext, false);
pub const FilesContext = struct {
pub fn hash(fc: FilesContext, file: File) u32 {
_ = fc;
return file.prefixed_path.hash();
}
pub fn eql(fc: FilesContext, a: File, b: File, b_index: usize) bool {
_ = fc;
_ = b_index;
return a.prefixed_path.eql(b.prefixed_path);
}
};
const FilesAdapter = struct {
pub fn eql(context: @This(), a: PrefixedPath, b: File, b_index: usize) bool {
_ = context;
_ = b_index;
return a.eql(b.prefixed_path);
}
pub fn hash(context: @This(), key: PrefixedPath) u32 {
_ = context;
return key.hash();
}
};
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// Add a file as a dependency of process being cached. When `hit` is
Add documentation to CacheHash API
2020-04-07 23:57:19 -06:00
/// called, the file's contents will be checked to ensure that it matches
/// the contents from previous times.
Return an index from `CacheHash.addFile` This makes it possible for the user to retrieve the contents of the file without running into data races.
2020-04-15 20:13:26 -06:00
///
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
/// Max file size will be used to determine the amount of space the file contents
Add max_file_size argument
2020-05-01 23:06:10 -06:00
/// are allowed to take up in memory. If max_file_size is null, then the contents
/// will not be loaded into memory.
///
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// Returns the index of the entry in the `files` array list. You can use it
/// to access the contents of the file after calling `hit()` like so:
Return an index from `CacheHash.addFile` This makes it possible for the user to retrieve the contents of the file without running into data races.
2020-04-15 20:13:26 -06:00
///
/// ```
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
/// var file_contents = cache_hash.files.keys()[file_index].contents.?;
Return an index from `CacheHash.addFile` This makes it possible for the user to retrieve the contents of the file without running into data races.
2020-04-15 20:13:26 -06:00
/// ```
introduce std.Build.Cache.Manifest.addFilePath and deprecate `addFile`. Part of an effort to move towards using `std.Build.Cache.Path` abstraction in more places, which makes it easier to avoid absolute paths and path resolution.
2024-07-10 15:08:23 -07:00
pub fn addFilePath(m: *Manifest, file_path: Path, max_file_size: ?usize) !usize {
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
return addOpenedFile(m, file_path, null, max_file_size);
}
/// Same as `addFilePath` except the file has already been opened.
pub fn addOpenedFile(m: *Manifest, path: Path, handle: ?fs.File, max_file_size: ?usize) !usize {
introduce std.Build.Cache.Manifest.addFilePath and deprecate `addFile`. Part of an effort to move towards using `std.Build.Cache.Path` abstraction in more places, which makes it easier to avoid absolute paths and path resolution.
2024-07-10 15:08:23 -07:00
const gpa = m.cache.gpa;
try m.files.ensureUnusedCapacity(gpa, 1);
const resolved_path = try fs.path.resolve(gpa, &.{
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
path.root_dir.path orelse ".",
path.subPathOrDot(),
introduce std.Build.Cache.Manifest.addFilePath and deprecate `addFile`. Part of an effort to move towards using `std.Build.Cache.Path` abstraction in more places, which makes it easier to avoid absolute paths and path resolution.
2024-07-10 15:08:23 -07:00
});
errdefer gpa.free(resolved_path);
const prefixed_path = try m.cache.findPrefixResolved(resolved_path);
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
return addFileInner(m, prefixed_path, handle, max_file_size);
introduce std.Build.Cache.Manifest.addFilePath and deprecate `addFile`. Part of an effort to move towards using `std.Build.Cache.Path` abstraction in more places, which makes it easier to avoid absolute paths and path resolution.
2024-07-10 15:08:23 -07:00
}
/// Deprecated; use `addFilePath`.
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn addFile(self: *Manifest, file_path: []const u8, max_file_size: ?usize) !usize {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
assert(self.manifest_file == null);
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
const gpa = self.cache.gpa;
try self.files.ensureUnusedCapacity(gpa, 1);
const prefixed_path = try self.cache.findPrefix(file_path);
errdefer gpa.free(prefixed_path.sub_path);
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
return addFileInner(self, prefixed_path, null, max_file_size);
introduce std.Build.Cache.Manifest.addFilePath and deprecate `addFile`. Part of an effort to move towards using `std.Build.Cache.Path` abstraction in more places, which makes it easier to avoid absolute paths and path resolution.
2024-07-10 15:08:23 -07:00
}
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
fn addFileInner(self: *Manifest, prefixed_path: PrefixedPath, handle: ?fs.File, max_file_size: ?usize) usize {
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const gop = self.files.getOrPutAssumeCapacityAdapted(prefixed_path, FilesAdapter{});
if (gop.found_existing) {
gop.key_ptr.updateMaxSize(max_file_size);
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
gop.key_ptr.updateHandle(handle);
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
return gop.index;
}
gop.key_ptr.* = .{
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
.prefixed_path = prefixed_path,
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
.contents = null,
.max_file_size = max_file_size,
.stat = undefined,
.bin_digest = undefined,
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
.handle = handle,
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
};
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
self.hash.add(prefixed_path.prefix);
self.hash.addBytes(prefixed_path.sub_path);
Return an index from `CacheHash.addFile` This makes it possible for the user to retrieve the contents of the file without running into data races.
2020-04-15 20:13:26 -06:00
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
return gop.index;
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
link: fix false positive crtbegin/crtend detection Embrace the Path abstraction, doing more operations based on directory handles rather than absolute file paths. Most of the diff noise here comes from this one. Fix sorting of crtbegin/crtend atoms. Previously it would look at all path components for those strings. Make the C runtime path detection partially a pure function, and move some logic to glibc.zig where it belongs.
2024-10-10 00:41:58 -07:00
/// Deprecated, use `addOptionalFilePath`.
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn addOptionalFile(self: *Manifest, optional_file_path: ?[]const u8) !void {
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
self.hash.add(optional_file_path != null);
const file_path = optional_file_path orelse return;
_ = try self.addFile(file_path, null);
}
link: fix false positive crtbegin/crtend detection Embrace the Path abstraction, doing more operations based on directory handles rather than absolute file paths. Most of the diff noise here comes from this one. Fix sorting of crtbegin/crtend atoms. Previously it would look at all path components for those strings. Make the C runtime path detection partially a pure function, and move some logic to glibc.zig where it belongs.
2024-10-10 00:41:58 -07:00
pub fn addOptionalFilePath(self: *Manifest, optional_file_path: ?Path) !void {
self.hash.add(optional_file_path != null);
const file_path = optional_file_path orelse return;
_ = try self.addFilePath(file_path, null);
}
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn addListOfFiles(self: *Manifest, list_of_files: []const []const u8) !void {
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
self.hash.add(list_of_files.len);
for (list_of_files) |file_path| {
_ = try self.addFile(file_path, null);
}
}
Build.Step.Run: fix cache management when there are side effects Closes #19947
2024-05-12 23:09:40 -04:00
pub fn addDepFile(self: *Manifest, dir: fs.Dir, dep_file_basename: []const u8) !void {
assert(self.manifest_file == null);
return self.addDepFileMaybePost(dir, dep_file_basename);
}
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
pub const HitError = error{
/// Unable to check the cache for a reason that has been recorded into
/// the `diagnostic` field.
CacheCheckFailed,
/// A cache manifest file exists however it could not be parsed.
InvalidFormat,
OutOfMemory,
};
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// Check the cache to see if the input exists in it. If it exists, returns `true`.
stage2 Cache: use hex instead of base64 for file paths
2020-09-16 12:31:42 -07:00
/// A hex encoding of its hash is available by calling `final`.
Add documentation to CacheHash API
2020-04-07 23:57:19 -06:00
///
/// This function will also acquire an exclusive lock to the manifest file. This means
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
/// that a process holding a Manifest will block any other process attempting to
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
/// acquire the lock. If `want_shared_lock` is `true`, a cache hit guarantees the
/// manifest file to be locked in shared mode, and a cache miss guarantees the manifest
/// file to be locked in exclusive mode.
Add documentation to CacheHash API
2020-04-07 23:57:19 -06:00
///
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// The lock on the manifest file is released when `deinit` is called. As another
/// option, one may call `toOwnedLock` to obtain a smaller object which can represent
/// the lock. `deinit` is safe to call whether or not `toOwnedLock` has been called.
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
pub fn hit(self: *Manifest) HitError!bool {
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
const gpa = self.cache.gpa;
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
assert(self.manifest_file == null);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
self.diagnostic = .none;
stage2: better error message for root zig source file not found closes #6777 closes #6893
2020-12-28 21:48:56 -07:00
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
const ext = ".txt";
Zir: implement explicit block_comptime instruction Resolves: #7056
2023-03-05 12:39:32 +00:00
var manifest_file_path: [hex_digest_len + ext.len]u8 = undefined;
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
var bin_digest: BinDigest = undefined;
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
self.hash.hasher.final(&bin_digest);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
std.Build.Cache: add binToHex function reduces need for API users to rely on formatted printing, even though that's how it is currently implemented.
2024-07-04 14:15:15 -07:00
self.hex_digest = binToHex(bin_digest);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
self.hash.hasher = hasher_init;
self.hash.hasher.update(&bin_digest);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
update codebase to use `@memset` and `@memcpy`
2023-04-26 13:57:08 -07:00
@memcpy(manifest_file_path[0..self.hex_digest.len], &self.hex_digest);
Zir: implement explicit block_comptime instruction Resolves: #7056
2023-03-05 12:39:32 +00:00
manifest_file_path[hex_digest_len..][0..ext.len].* = ext.*;
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
while (true) {
if (self.cache.manifest_dir.createFile(&manifest_file_path, .{
.read = true,
.truncate = false,
std.fs.file: Rename File.Lock enum values to snake case
2023-05-20 23:11:53 +01:00
.lock = .exclusive,
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
.lock_nonblocking = self.want_shared_lock,
})) |manifest_file| {
self.manifest_file = manifest_file;
self.have_exclusive_lock = true;
break;
} else |err| switch (err) {
error.WouldBlock => {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
self.manifest_file = self.cache.manifest_dir.openFile(&manifest_file_path, .{
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
.mode = .read_write,
std.fs.file: Rename File.Lock enum values to snake case
2023-05-20 23:11:53 +01:00
.lock = .shared,
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
}) catch |e| {
self.diagnostic = .{ .manifest_create = e };
return error.CacheCheckFailed;
};
Cache: fix multi-process race condition on macOS This fixes `.INVAL => unreachable` being triggered by the cache system on macOS when multiple processes race to create the same compilation. The problem is that when two processes race to create a file, it sometimes returns ENOENT even though that error code is nonsensical for this situation. Commit 2b0929929d67e222ca6a9523a3a594ed456c4a51 purportedly solved this, but it did not open the file with write permissions, leading to the EINVAL panic later on. This commit remedies the situation by introducing a loop and simply retrying when the ENOENT occurs.
2023-04-18 13:08:30 -07:00
break;
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
},
std.Build.Cache.hit: work around macOS kernel bug The previous commit cast doubt upon the initial report about macOS kernel behavior, identifying another reason that ENOENT could be returned from file creation. However, it is demonstrable that ENOENT can be returned for both cases: 1. create file race 2. handle refers to deleted directory This commit re-introduces the workaround for the file creation race on macOS however it does not unconditionally retry - it first tries again with O_EXCL to disambiguate the error condition that has occurred.
2024-12-10 20:44:00 -08:00
error.FileNotFound => {
// There are no dir components, so the only possibility
// should be that the directory behind the handle has been
// deleted, however we have observed on macOS two processes
// racing to do openat() with O_CREAT manifest in ENOENT.
//
// As a workaround, we retry with exclusive=true which
// disambiguates by returning EEXIST, indicating original
// failure was a race, or ENOENT, indicating deletion of
// the directory of our open handle.
if (builtin.os.tag != .macos) {
self.diagnostic = .{ .manifest_create = error.FileNotFound };
return error.CacheCheckFailed;
}
if (self.cache.manifest_dir.createFile(&manifest_file_path, .{
.read = true,
.truncate = false,
.lock = .exclusive,
.lock_nonblocking = self.want_shared_lock,
.exclusive = true,
})) |manifest_file| {
self.manifest_file = manifest_file;
self.have_exclusive_lock = true;
break;
} else |excl_err| switch (excl_err) {
error.WouldBlock, error.PathAlreadyExists => continue,
error.FileNotFound => {
self.diagnostic = .{ .manifest_create = error.FileNotFound };
return error.CacheCheckFailed;
},
else => |e| {
self.diagnostic = .{ .manifest_create = e };
return error.CacheCheckFailed;
},
}
},
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
else => |e| {
self.diagnostic = .{ .manifest_create = e };
return error.CacheCheckFailed;
},
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
Cache: use mutex to protect recent_problematic_timestamp The previous commit tried to use atomics but not many CPUs support 128-bit atomics. So we use a mutex. In order to avoid contention, we also store `recent_problematic_timestamp` locally on the `Manifest` which is only ever accessed from a single thread at a time, and only consult the global one if the local one is problematic. This commit was tested by running `zig build test-behavior` in two separate terminals at the same time.
2021-12-09 22:07:28 -07:00
self.want_refresh_timestamp = true;
Cache: fix data race with is_problematic_timestamp Previously `recent_problematic_timestamp` was unprotected and accessed potentially with multiple worker threads simultaneously. This commit protects it with atomics and also introduces a flag to prevent multiple timestamp checks from within the same call to hit(). Unfortunately the compiler-rt function __sync_val_compare_and_swap_16 is not yet implemented, so I will have to take a different strategy in a follow-up commit.
2021-12-09 21:14:39 -07:00
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const input_file_count = self.files.entries.len;
Cache: fix logic for retrying cache hits Fixes potentially #16149
2024-02-04 03:46:11 +01:00
while (true) : (self.unhit(bin_digest, input_file_count)) {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
const file_contents = self.manifest_file.?.reader().readAllAlloc(gpa, manifest_file_size_max) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
error.StreamTooLong => return error.OutOfMemory,
else => |e| {
self.diagnostic = .{ .manifest_read = e };
return error.CacheCheckFailed;
},
};
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
defer gpa.free(file_contents);
var any_file_changed = false;
Update all std.mem.tokenize calls to their appropriate function Everywhere that can now use `tokenizeScalar` should get a nice little performance boost.
2023-05-04 18:05:40 -07:00
var line_iter = mem.tokenizeScalar(u8, file_contents, '\n');
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
var idx: usize = 0;
if (if (line_iter.next()) |line| !std.mem.eql(u8, line, manifest_header) else true) {
if (try self.upgradeToExclusiveLock()) continue;
self.manifest_dirty = true;
while (idx < input_file_count) : (idx += 1) {
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const ch_file = &self.files.keys()[idx];
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
self.populateFileHash(ch_file) catch |err| {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
self.diagnostic = .{ .file_hash = .{
.file_index = idx,
.err = err,
} };
return error.CacheCheckFailed;
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
};
}
return false;
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
}
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
while (line_iter.next()) |line| {
defer idx += 1;
Update all std.mem.tokenize calls to their appropriate function Everywhere that can now use `tokenizeScalar` should get a nice little performance boost.
2023-05-04 18:05:40 -07:00
var iter = mem.tokenizeScalar(u8, line, ' ');
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
const size = iter.next() orelse return error.InvalidFormat;
const inode = iter.next() orelse return error.InvalidFormat;
const mtime_nsec_str = iter.next() orelse return error.InvalidFormat;
const digest_str = iter.next() orelse return error.InvalidFormat;
const prefix_str = iter.next() orelse return error.InvalidFormat;
const file_path = iter.rest();
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const stat_size = fmt.parseInt(u64, size, 10) catch return error.InvalidFormat;
const stat_inode = fmt.parseInt(fs.File.INode, inode, 10) catch return error.InvalidFormat;
const stat_mtime = fmt.parseInt(i64, mtime_nsec_str, 10) catch return error.InvalidFormat;
const file_bin_digest = b: {
if (digest_str.len != hex_digest_len) return error.InvalidFormat;
var bd: BinDigest = undefined;
_ = fmt.hexToBytes(&bd, digest_str) catch return error.InvalidFormat;
break :b bd;
};
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
const prefix = fmt.parseInt(u8, prefix_str, 10) catch return error.InvalidFormat;
if (prefix >= self.cache.prefixes_len) return error.InvalidFormat;
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
if (file_path.len == 0) return error.InvalidFormat;
Remove file handle from CacheHash A file handle is not the same thing as an inode index number. Eventually the inode will be checked as well, but there needs to be a way to get the inode in `std` first.
2020-03-08 15:11:06 -06:00
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const cache_hash_file = f: {
const prefixed_path: PrefixedPath = .{
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
.prefix = prefix,
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
.sub_path = file_path, // expires with file_contents
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
};
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
if (idx < input_file_count) {
const file = &self.files.keys()[idx];
if (!file.prefixed_path.eql(prefixed_path))
return error.InvalidFormat;
file.stat = .{
.size = stat_size,
.inode = stat_inode,
.mtime = stat_mtime,
};
file.bin_digest = file_bin_digest;
break :f file;
}
const gop = try self.files.getOrPutAdapted(gpa, prefixed_path, FilesAdapter{});
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
errdefer _ = self.files.pop();
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
if (!gop.found_existing) {
gop.key_ptr.* = .{
.prefixed_path = .{
.prefix = prefix,
.sub_path = try gpa.dupe(u8, file_path),
},
.contents = null,
.max_file_size = null,
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
.handle = null,
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
.stat = .{
.size = stat_size,
.inode = stat_inode,
.mtime = stat_mtime,
},
.bin_digest = file_bin_digest,
};
}
break :f gop.key_ptr;
};
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const pp = cache_hash_file.prefixed_path;
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
const dir = self.cache.prefixes()[pp.prefix].handle;
const this_file = dir.openFile(pp.sub_path, .{ .mode = .read_only }) catch |err| switch (err) {
error.FileNotFound => {
if (try self.upgradeToExclusiveLock()) continue;
return false;
},
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
else => |e| {
self.diagnostic = .{ .file_open = .{
.file_index = idx,
.err = e,
} };
return error.CacheCheckFailed;
},
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
};
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
defer this_file.close();
Remove file handle from CacheHash A file handle is not the same thing as an inode index number. Eventually the inode will be checked as well, but there needs to be a way to get the inode in `std` first.
2020-03-08 15:11:06 -06:00
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
const actual_stat = this_file.stat() catch |err| {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
self.diagnostic = .{ .file_stat = .{
.file_index = idx,
.err = err,
} };
return error.CacheCheckFailed;
stage2: better error message for root zig source file not found closes #6777 closes #6893
2020-12-28 21:48:56 -07:00
};
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
const size_match = actual_stat.size == cache_hash_file.stat.size;
const mtime_match = actual_stat.mtime == cache_hash_file.stat.mtime;
const inode_match = actual_stat.inode == cache_hash_file.stat.inode;
if (!size_match or !mtime_match or !inode_match) {
self.manifest_dirty = true;
cache_hash_file.stat = .{
.size = actual_stat.size,
.mtime = actual_stat.mtime,
.inode = actual_stat.inode,
};
if (self.isProblematicTimestamp(cache_hash_file.stat.mtime)) {
// The actual file has an unreliable timestamp, force it to be hashed
cache_hash_file.stat.mtime = 0;
cache_hash_file.stat.inode = 0;
}
var actual_digest: BinDigest = undefined;
hashFile(this_file, &actual_digest) catch |err| {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
self.diagnostic = .{ .file_read = .{
.file_index = idx,
.err = err,
} };
return error.CacheCheckFailed;
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
};
if (!mem.eql(u8, &cache_hash_file.bin_digest, &actual_digest)) {
cache_hash_file.bin_digest = actual_digest;
// keep going until we have the input file digests
any_file_changed = true;
}
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
if (!any_file_changed) {
self.hash.hasher.update(&cache_hash_file.bin_digest);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
}
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
if (any_file_changed) {
if (try self.upgradeToExclusiveLock()) continue;
// cache miss
// keep the manifest file open
self.unhit(bin_digest, input_file_count);
return false;
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
if (idx < input_file_count) {
if (try self.upgradeToExclusiveLock()) continue;
self.manifest_dirty = true;
while (idx < input_file_count) : (idx += 1) {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
self.populateFileHash(&self.files.keys()[idx]) catch |err| {
self.diagnostic = .{ .file_hash = .{
.file_index = idx,
.err = err,
} };
return error.CacheCheckFailed;
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
};
}
return false;
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
if (self.want_shared_lock) {
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
self.downgradeToSharedLock() catch |err| {
self.diagnostic = .{ .manifest_lock = err };
return error.CacheCheckFailed;
};
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
return true;
cache: Fix LockViolation during C compilation paths (#13591) - C compilation flows didn't hold an exclusive lock on the cache manifest file when writing to it in all cases - On windows, explicitly unlock the file lock before closing it
2022-12-06 23:15:54 -05:00
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
pub fn unhit(self: *Manifest, bin_digest: BinDigest, input_file_count: usize) void {
stage2: implement zig build As part of this: * add std.process.cleanExit. closes #6395 - use it in several places * adjust the alignment of text in `zig build --help` menu * Cache: support the concept of "unhit" so that we properly keep track of the cache when we find out using the secondary hash that the cache "hit" was actually a miss. Use this to fix false negatives of caching of stage1 build artifacts. * fix not deleting the symlink hash for stage1 build artifacts causing false positives. * implement support for Package arguments in stage1 build artifacts * update and add missing usage text * add --override-lib-dir and --enable-cache CLI options - `--enable-cache` takes the place of `--cache on` * CLI supports -femit-bin=foo combined with --enable-cache to do an "update file" operation. --enable-cache without that argument will build the output into a cache directory and then print the path to stdout (matching master branch behavior). * errors surfacing from main() now print "error: Foo" instead of "error: error.Foo".
2020-09-22 22:18:19 -07:00
// Reset the hash.
self.hash.hasher = hasher_init;
self.hash.hasher.update(&bin_digest);
// Remove files not in the initial hash.
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
while (self.files.count() != input_file_count) {
std.ArrayHashMap: popOrNul() -> pop()
2025-02-02 00:03:19 -08:00
var file = self.files.pop().?;
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
file.key.deinit(self.cache.gpa);
stage2: implement zig build As part of this: * add std.process.cleanExit. closes #6395 - use it in several places * adjust the alignment of text in `zig build --help` menu * Cache: support the concept of "unhit" so that we properly keep track of the cache when we find out using the secondary hash that the cache "hit" was actually a miss. Use this to fix false negatives of caching of stage1 build artifacts. * fix not deleting the symlink hash for stage1 build artifacts causing false positives. * implement support for Package arguments in stage1 build artifacts * update and add missing usage text * add --override-lib-dir and --enable-cache CLI options - `--enable-cache` takes the place of `--cache on` * CLI supports -femit-bin=foo combined with --enable-cache to do an "update file" operation. --enable-cache without that argument will build the output into a cache directory and then print the path to stdout (matching master branch behavior). * errors surfacing from main() now print "error: Foo" instead of "error: error.Foo".
2020-09-22 22:18:19 -07:00
}
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
for (self.files.keys()) |file| {
stage2: implement zig build As part of this: * add std.process.cleanExit. closes #6395 - use it in several places * adjust the alignment of text in `zig build --help` menu * Cache: support the concept of "unhit" so that we properly keep track of the cache when we find out using the secondary hash that the cache "hit" was actually a miss. Use this to fix false negatives of caching of stage1 build artifacts. * fix not deleting the symlink hash for stage1 build artifacts causing false positives. * implement support for Package arguments in stage1 build artifacts * update and add missing usage text * add --override-lib-dir and --enable-cache CLI options - `--enable-cache` takes the place of `--cache on` * CLI supports -femit-bin=foo combined with --enable-cache to do an "update file" operation. --enable-cache without that argument will build the output into a cache directory and then print the path to stdout (matching master branch behavior). * errors surfacing from main() now print "error: Foo" instead of "error: error.Foo".
2020-09-22 22:18:19 -07:00
self.hash.hasher.update(&file.bin_digest);
}
}
Cache: use mutex to protect recent_problematic_timestamp The previous commit tried to use atomics but not many CPUs support 128-bit atomics. So we use a mutex. In order to avoid contention, we also store `recent_problematic_timestamp` locally on the `Manifest` which is only ever accessed from a single thread at a time, and only consult the global one if the local one is problematic. This commit was tested by running `zig build test-behavior` in two separate terminals at the same time.
2021-12-09 22:07:28 -07:00
fn isProblematicTimestamp(man: *Manifest, file_time: i128) bool {
// If the file_time is prior to the most recent problematic timestamp
// then we don't need to access the filesystem.
if (file_time < man.recent_problematic_timestamp)
return false;
// Next we will check the globally shared Cache timestamp, which is accessed
// from multiple threads.
man.cache.mutex.lock();
defer man.cache.mutex.unlock();
// Save the global one to our local one to avoid locking next time.
man.recent_problematic_timestamp = man.cache.recent_problematic_timestamp;
if (file_time < man.recent_problematic_timestamp)
return false;
// This flag prevents multiple filesystem writes for the same hit() call.
if (man.want_refresh_timestamp) {
man.want_refresh_timestamp = false;
var file = man.cache.manifest_dir.createFile("timestamp", .{
.read = true,
.truncate = true,
}) catch return true;
defer file.close();
// Save locally and also save globally (we still hold the global lock).
man.recent_problematic_timestamp = (file.stat() catch return true).mtime;
man.cache.recent_problematic_timestamp = man.recent_problematic_timestamp;
}
return file_time >= man.recent_problematic_timestamp;
}
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
fn populateFileHash(self: *Manifest, ch_file: *File) !void {
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
if (ch_file.handle) |handle| {
return populateFileHashHandle(self, ch_file, handle);
} else {
const pp = ch_file.prefixed_path;
const dir = self.cache.prefixes()[pp.prefix].handle;
const handle = try dir.openFile(pp.sub_path, .{});
defer handle.close();
return populateFileHashHandle(self, ch_file, handle);
}
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
fn populateFileHashHandle(self: *Manifest, ch_file: *File, handle: fs.File) !void {
const actual_stat = try handle.stat();
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
ch_file.stat = .{
.size = actual_stat.size,
.mtime = actual_stat.mtime,
.inode = actual_stat.inode,
};
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Cache: use mutex to protect recent_problematic_timestamp The previous commit tried to use atomics but not many CPUs support 128-bit atomics. So we use a mutex. In order to avoid contention, we also store `recent_problematic_timestamp` locally on the `Manifest` which is only ever accessed from a single thread at a time, and only consult the global one if the local one is problematic. This commit was tested by running `zig build test-behavior` in two separate terminals at the same time.
2021-12-09 22:07:28 -07:00
if (self.isProblematicTimestamp(ch_file.stat.mtime)) {
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
// The actual file has an unreliable timestamp, force it to be hashed
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
ch_file.stat.mtime = 0;
ch_file.stat.inode = 0;
Check for problematic timestamps
2020-04-11 16:01:17 -06:00
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
if (ch_file.max_file_size) |max_file_size| {
if (ch_file.stat.size > max_file_size) {
return error.FileTooBig;
}
Add `addFilePost` and `addFilePostFetch` functions
2020-04-14 19:33:02 -06:00
all: migrate code to new cast builtin syntax Most of this migration was performed automatically with `zig fmt`. There were a few exceptions which I had to manually fix: * `@alignCast` and `@addrSpaceCast` cannot be automatically rewritten * `@truncate`'s fixup is incorrect for vectors * Test cases are not formatted, and their error locations change
2023-06-22 18:46:56 +01:00
const contents = try self.cache.gpa.alloc(u8, @as(usize, @intCast(ch_file.stat.size)));
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
errdefer self.cache.gpa.free(contents);
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
// Hash while reading from disk, to keep the contents in the cpu cache while
// doing hashing.
stage2: more progress moving `zig cc` to stage2 * std.cache_hash exposes Hasher type * std.cache_hash makes hasher_init a global const * std.cache_hash supports cloning so that clones can share the same open manifest dir handle as well as fork from shared hasher state * start to populate the cache_hash for stage2 builds * remove a footgun from std.cache_hash add function * get rid of std.Target.ObjectFormat.unknown * rework stage2 logic for resolving output artifact names by adding object_format as an optional parameter to std.zig.binNameAlloc * support -Denable-llvm in stage2 tests * Module supports the use case when there are no .zig files * introduce c_object_table and failed_c_objects to Module * propagate many new kinds of data from CLI into Module and into linker.Options * introduce -fLLVM, -fLLD, -fClang and their -fno- counterparts. closes #6251. - add logic for choosing when to use LLD or zig's self-hosted linker * stub code for implementing invoking Clang to build C objects * add -femit-h, -femit-h=foo, and -fno-emit-h CLI options
2020-09-08 01:11:10 -07:00
var hasher = hasher_init;
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
var off: usize = 0;
while (true) {
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
const bytes_read = try handle.pread(contents[off..], off);
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
if (bytes_read == 0) break;
cache_hash: hash function change This makes the `cache_hash` hash function easier to replace. BLAKE3 would be a natural fit for hashing large files, but: - second preimage resistance is not necessary for the cache_hash use cases - our BLAKE3 implementation is currently very slow Switch to SipHash128, which gives us an immediate speed boost.
2020-08-21 15:08:15 +02:00
hasher.update(contents[off..][0..bytes_read]);
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
off += bytes_read;
}
cache_hash: hash function change This makes the `cache_hash` hash function easier to replace. BLAKE3 would be a natural fit for hashing large files, but: - second preimage resistance is not necessary for the cache_hash use cases - our BLAKE3 implementation is currently very slow Switch to SipHash128, which gives us an immediate speed boost.
2020-08-21 15:08:15 +02:00
hasher.final(&ch_file.bin_digest);
Add `addFilePost` and `addFilePostFetch` functions
2020-04-14 19:33:02 -06:00
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
ch_file.contents = contents;
} else {
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
try hashFile(handle, &ch_file.bin_digest);
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
}
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
self.hash.hasher.update(&ch_file.bin_digest);
Add `addFilePost` and `addFilePostFetch` functions
2020-04-14 19:33:02 -06:00
}
/// Add a file as a dependency of process being cached, after the initial hash has been
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
/// calculated. This is useful for processes that don't know all the files that
Make `addFilePost*` functions' documentation more clear
2020-04-14 22:17:55 -06:00
/// are depended on ahead of time. For example, a source file that can import other files
/// will need to be recompiled if the imported file is changed.
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn addFilePostFetch(self: *Manifest, file_path: []const u8, max_file_size: usize) ![]const u8 {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
assert(self.manifest_file != null);
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
const gpa = self.cache.gpa;
const prefixed_path = try self.cache.findPrefix(file_path);
errdefer gpa.free(prefixed_path.sub_path);
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
const gop = try self.files.getOrPutAdapted(gpa, prefixed_path, FilesAdapter{});
errdefer _ = self.files.pop();
if (gop.found_existing) {
gpa.free(prefixed_path.sub_path);
return gop.key_ptr.contents.?;
}
gop.key_ptr.* = .{
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
.prefixed_path = prefixed_path,
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
.max_file_size = max_file_size,
.stat = undefined,
.bin_digest = undefined,
.contents = null,
};
Add `addFilePost` and `addFilePostFetch` functions
2020-04-14 19:33:02 -06:00
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
self.files.lockPointers();
defer self.files.unlockPointers();
Add max_file_size argument
2020-05-01 23:06:10 -06:00
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
try self.populateFileHash(gop.key_ptr);
return gop.key_ptr.contents.?;
Add `addFilePost` and `addFilePostFetch` functions
2020-04-14 19:33:02 -06:00
}
/// Add a file as a dependency of process being cached, after the initial hash has been
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
/// calculated.
///
/// This is useful for processes that don't know the all the files that are
/// depended on ahead of time. For example, a source file that can import
/// other files will need to be recompiled if the imported file is changed.
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn addFilePost(self: *Manifest, file_path: []const u8) !void {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
assert(self.manifest_file != null);
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
const gpa = self.cache.gpa;
const prefixed_path = try self.cache.findPrefix(file_path);
errdefer gpa.free(prefixed_path.sub_path);
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const gop = try self.files.getOrPutAdapted(gpa, prefixed_path, FilesAdapter{});
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
errdefer _ = self.files.pop();
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
if (gop.found_existing) {
gpa.free(prefixed_path.sub_path);
return;
}
gop.key_ptr.* = .{
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
.prefixed_path = prefixed_path,
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
.max_file_size = null,
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
.handle = null,
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
.stat = undefined,
.bin_digest = undefined,
.contents = null,
};
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
self.files.lockPointers();
defer self.files.unlockPointers();
try self.populateFileHash(gop.key_ptr);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
/// Like `addFilePost` but when the file contents have already been loaded from disk.
/// On success, cache takes ownership of `resolved_path`.
pub fn addFilePostContents(
self: *Manifest,
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
resolved_path: []u8,
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
bytes: []const u8,
stat: File.Stat,
Cache: Fix findPrefix when paths are slightly out of the ordinary This makes Cache.findPrefix/findPrefixResolved use `std.fs.path.relative` instead of `std.mem.startsWith` when checking if a file is within a prefix. This fixes multiple edge cases around prefix detection: - If a prefix path ended with a path separator, then the first character of the 'sub_path' would get cut off because the previous implementation assumed it was a path separator. Example: prefix: `/foo/`, file_path: `/foo/abc.txt` would see that they both start with `/foo/` and then slice starting from one byte past the common prefix, ending up with `bc.txt` instead of the expected `abc.txt` - If a prefix contained double path separators after any component, then the `startsWith` check would erroneously fail. Example: prefix: `/foo//bar`, file_path: `/foo/bar/abc.txt` would not see that abc.txt is a sub path of the prefix `/foo//bar` - On Windows, case insensitivity was not respected at all, instead the UTF-8 bytes were compared directly This fixes all of the things in the above list (and possibly more).
2023-08-19 15:41:09 -07:00
) !void {
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
assert(self.manifest_file != null);
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
const gpa = self.cache.gpa;
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
const prefixed_path = try self.cache.findPrefixResolved(resolved_path);
errdefer gpa.free(prefixed_path.sub_path);
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
const gop = try self.files.getOrPutAdapted(gpa, prefixed_path, FilesAdapter{});
Build.Cache: fix UAF during `unhit`
2024-03-23 03:58:32 +01:00
errdefer _ = self.files.pop();
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
if (gop.found_existing) {
gpa.free(prefixed_path.sub_path);
return;
}
const new_file = gop.key_ptr;
new_file.* = .{
Cache: introduce prefixes to manifests Before, cache manifest files would have absolute file paths. This is problematic for two reasons: * Absolute file paths are not portable. Some operating systems such as WASI have trouble with them. The files themselves are less portable; they cannot be migrated from one user's home directory to another's. And finally they can break due to file paths exceeding maximum path component size. * They would prevent some advanced use cases of Zig, where the lib dir has a different path in a different invocation but is ultimately the same Zig version and lib directory as before. This commit adds a new column that specifies the prefix directory for each file. 0 is an escape hatch and has the previous behavior. The other two prefixes introduced are zig lib directory, and the cache directory. This means files in zig-cache manifests can reference files local to these directories. In practice, this means it is possible to use a different file path for the zig lib directory in a subsequent run of zig and have it still take advantage of the global cache, provided that the files inside remain unchanged. closes #13050
2022-11-19 13:48:32 -07:00
.prefixed_path = prefixed_path,
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
.max_file_size = null,
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
.handle = null,
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
.stat = stat,
.bin_digest = undefined,
.contents = null,
};
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
if (self.isProblematicTimestamp(new_file.stat.mtime)) {
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
// The actual file has an unreliable timestamp, force it to be hashed
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
new_file.stat.mtime = 0;
new_file.stat.inode = 0;
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
}
{
var hasher = hasher_init;
hasher.update(bytes);
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
hasher.final(&new_file.bin_digest);
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
}
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
self.hash.hasher.update(&new_file.bin_digest);
stage2: add `@import` and `@embedFile` to CacheHash when using `CacheMode.whole`. Also, I verified that `addDepFilePost` is in fact including the original C source file in addition to the files it depends on.
2021-12-30 16:42:32 -07:00
}
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn addDepFilePost(self: *Manifest, dir: fs.Dir, dep_file_basename: []const u8) !void {
stage2: implement .d file parsing for C objects
2020-09-15 18:02:42 -07:00
assert(self.manifest_file != null);
Build.Step.Run: fix cache management when there are side effects Closes #19947
2024-05-12 23:09:40 -04:00
return self.addDepFileMaybePost(dir, dep_file_basename);
}
stage2: implement .d file parsing for C objects
2020-09-15 18:02:42 -07:00
Build.Step.Run: fix cache management when there are side effects Closes #19947
2024-05-12 23:09:40 -04:00
fn addDepFileMaybePost(self: *Manifest, dir: fs.Dir, dep_file_basename: []const u8) !void {
stage2 Cache: use hex instead of base64 for file paths
2020-09-16 12:31:42 -07:00
const dep_file_contents = try dir.readFileAlloc(self.cache.gpa, dep_file_basename, manifest_file_size_max);
stage2: implement .d file parsing for C objects
2020-09-15 18:02:42 -07:00
defer self.cache.gpa.free(dep_file_contents);
stage2: update uses of DepTokenizer
2020-09-19 14:16:58 +03:00
var error_buf = std.ArrayList(u8).init(self.cache.gpa);
defer error_buf.deinit();
move the cache system from compiler to std lib
2023-02-05 19:39:04 -07:00
var it: DepTokenizer = .{ .bytes = dep_file_contents };
stage2: implement .d file parsing for C objects
2020-09-15 18:02:42 -07:00
Build system: Support Windows depfiles with unquoted, backslash escaped spaces (#20100)
2024-06-06 13:40:10 -05:00
while (it.next()) |token| {
switch (token) {
std.Build: add support for deps .d file in Step.Run
2023-08-12 13:15:05 +02:00
// We don't care about targets, we only want the prereqs
// Clang is invoked in single-source mode but other programs may not
.target, .target_must_resolve => {},
Build.Step.Run: fix cache management when there are side effects Closes #19947
2024-05-12 23:09:40 -04:00
.prereq => |file_path| if (self.manifest_file == null) {
_ = try self.addFile(file_path, null);
} else try self.addFilePost(file_path),
Build system: Support Windows depfiles with unquoted, backslash escaped spaces (#20100)
2024-06-06 13:40:10 -05:00
.prereq_must_resolve => {
var resolve_buf = std.ArrayList(u8).init(self.cache.gpa);
defer resolve_buf.deinit();
try token.resolve(resolve_buf.writer());
if (self.manifest_file == null) {
_ = try self.addFile(resolve_buf.items, null);
} else try self.addFilePost(resolve_buf.items);
},
stage2: update uses of DepTokenizer
2020-09-19 14:16:58 +03:00
else => |err| {
try err.printError(error_buf.writer());
Cache: add debug log statement
2021-11-24 23:08:37 -07:00
log.err("failed parsing {s}: {s}", .{ dep_file_basename, error_buf.items });
stage2: implement .d file parsing for C objects
2020-09-15 18:02:42 -07:00
return error.InvalidDepFile;
},
}
}
}
fix various issues related to Path handling in the compiler and std A compilation build step for which the binary is not required could not be compiled previously. There were 2 issues that caused this: - The compiler communicated only the results of the emitted binary and did not properly communicate the result if the binary was not emitted. This is fixed by communicating the final hash of the artifact path (the hash of the corresponding /o/<hash> directory) and communicating this instead of the entire path. This changes the zig build --listen protocol to communicate hashes instead of paths, and emit_bin_path is accordingly renamed to emit_digest. - There was an error related to the default llvm object path when CacheUse.Whole was selected. I'm not really sure why this didn't manifest when the binary is also emitted. This was fixed by improving the path handling related to flush() and emitLlvmObject(). In general, this commit also improves some of the path handling throughout the compiler and standard library.
2024-08-18 00:43:33 +02:00
/// Returns a binary hash of the inputs.
pub fn finalBin(self: *Manifest) BinDigest {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
assert(self.manifest_file != null);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
Add documentation to CacheHash API
2020-04-07 23:57:19 -06:00
// We don't close the manifest file yet, because we want to
// keep it locked until the API user is done using it.
// We also don't write out the manifest yet, because until
// cache_release is called we still might be working on creating
// the artifacts to cache.
stage2: detect redundant C/C++ source files Cache exposes BinDigest. Compilation gains a set of a BinDigest for every C/C++ source file. We detect when the same source/flags have already been added and emit a compile error. This prevents a deadlock in the caching system. Closes #7308
2020-12-10 21:12:05 -07:00
var bin_digest: BinDigest = undefined;
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
self.hash.hasher.final(&bin_digest);
fix various issues related to Path handling in the compiler and std A compilation build step for which the binary is not required could not be compiled previously. There were 2 issues that caused this: - The compiler communicated only the results of the emitted binary and did not properly communicate the result if the binary was not emitted. This is fixed by communicating the final hash of the artifact path (the hash of the corresponding /o/<hash> directory) and communicating this instead of the entire path. This changes the zig build --listen protocol to communicate hashes instead of paths, and emit_bin_path is accordingly renamed to emit_digest. - There was an error related to the default llvm object path when CacheUse.Whole was selected. I'm not really sure why this didn't manifest when the binary is also emitted. This was fixed by improving the path handling related to flush() and emitLlvmObject(). In general, this commit also improves some of the path handling throughout the compiler and standard library.
2024-08-18 00:43:33 +02:00
return bin_digest;
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
fix various issues related to Path handling in the compiler and std A compilation build step for which the binary is not required could not be compiled previously. There were 2 issues that caused this: - The compiler communicated only the results of the emitted binary and did not properly communicate the result if the binary was not emitted. This is fixed by communicating the final hash of the artifact path (the hash of the corresponding /o/<hash> directory) and communicating this instead of the entire path. This changes the zig build --listen protocol to communicate hashes instead of paths, and emit_bin_path is accordingly renamed to emit_digest. - There was an error related to the default llvm object path when CacheUse.Whole was selected. I'm not really sure why this didn't manifest when the binary is also emitted. This was fixed by improving the path handling related to flush() and emitLlvmObject(). In general, this commit also improves some of the path handling throughout the compiler and standard library.
2024-08-18 00:43:33 +02:00
/// Returns a hex encoded hash of the inputs.
pub fn final(self: *Manifest) HexDigest {
const bin_digest = self.finalBin();
std.Build.Cache: add binToHex function reduces need for API users to rely on formatted printing, even though that's how it is currently implemented.
2024-07-04 14:15:15 -07:00
return binToHex(bin_digest);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
/// If `want_shared_lock` is true, this function automatically downgrades the
/// lock from exclusive to shared.
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn writeManifest(self: *Manifest) !void {
Fix another LockViolation case on Windows (#14162) - Add an assert that an exclusive lock is help to writeManifest - Only call writeManifest in updateCObject if an exclusive lock is held - cache: fixup test to verify hits don't take an exclusive lock, instead of writing the manifest
2023-01-04 14:51:43 -05:00
assert(self.have_exclusive_lock);
stage2: fix Cache not calling ftruncate in writeManifest this led to a corrupt cache when the number of files got smaller. it is now fixed.
2020-09-28 22:40:50 -07:00
const manifest_file = self.manifest_file.?;
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
if (self.manifest_dirty) {
self.manifest_dirty = false;
var contents = std.ArrayList(u8).init(self.cache.gpa);
defer contents.deinit();
const writer = contents.writer();
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
try writer.writeAll(manifest_header ++ "\n");
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
for (self.files.keys()) |file| {
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
try writer.print("{d} {d} {d} {} {d} {s}\n", .{
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
file.stat.size,
file.stat.inode,
file.stat.mtime,
Cache: fix race condition When checking a cache entry with no input files for a hit, if `createFile` returned `error.WouldBlock` we would forget about the fact that the file has been created, and all future checks will assume that a cache hit has happened, even though one never has or does, leading to rare `FileNotFound` errors trying the access the protected files. This fix works by writing an extra byte to the manifest file to distinguish hits and misses when there no input files to write.
2023-05-10 01:52:33 -04:00
fmt.fmtSliceHexLower(&file.bin_digest),
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
file.prefixed_path.prefix,
file.prefixed_path.sub_path,
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
});
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
try manifest_file.setEndPos(contents.items.len);
try manifest_file.pwriteAll(contents.items, 0);
}
stage2: fix Cache not calling ftruncate in writeManifest this led to a corrupt cache when the number of files got smaller. it is now fixed.
2020-09-28 22:40:50 -07:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
if (self.want_shared_lock) {
try self.downgradeToSharedLock();
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
}
fn downgradeToSharedLock(self: *Manifest) !void {
if (!self.have_exclusive_lock) return;
stage2: Only bypass `flock` on WASI
2022-04-18 23:08:00 -07:00
// WASI does not currently support flock, so we bypass it here.
// TODO: If/when flock is supported on WASI, this check should be removed.
// See https://github.com/WebAssembly/wasi-filesystem/issues/2
if (builtin.os.tag != .wasi or std.process.can_spawn or !builtin.single_threaded) {
stage2: Bypass file locks in src/Cache.zig for WASI targets
2022-03-01 10:42:07 -07:00
const manifest_file = self.manifest_file.?;
try manifest_file.downgradeLock();
}
cache: Fix LockViolation during C compilation paths (#13591) - C compilation flows didn't hold an exclusive lock on the cache manifest file when writing to it in all cases - On windows, explicitly unlock the file lock before closing it
2022-12-06 23:15:54 -05:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
self.have_exclusive_lock = false;
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
fn upgradeToExclusiveLock(self: *Manifest) error{CacheCheckFailed}!bool {
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
if (self.have_exclusive_lock) return false;
cache: Fix LockViolation during C compilation paths (#13591) - C compilation flows didn't hold an exclusive lock on the cache manifest file when writing to it in all cases - On windows, explicitly unlock the file lock before closing it
2022-12-06 23:15:54 -05:00
assert(self.manifest_file != null);
stage2: Only bypass `flock` on WASI
2022-04-18 23:08:00 -07:00
// WASI does not currently support flock, so we bypass it here.
// TODO: If/when flock is supported on WASI, this check should be removed.
// See https://github.com/WebAssembly/wasi-filesystem/issues/2
if (builtin.os.tag != .wasi or std.process.can_spawn or !builtin.single_threaded) {
stage2: Bypass file locks in src/Cache.zig for WASI targets
2022-03-01 10:42:07 -07:00
const manifest_file = self.manifest_file.?;
// Here we intentionally have a period where the lock is released, in case there are
// other processes holding a shared lock.
manifest_file.unlock();
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
manifest_file.lock(.exclusive) catch |err| {
self.diagnostic = .{ .manifest_lock = err };
return error.CacheCheckFailed;
};
stage2: Bypass file locks in src/Cache.zig for WASI targets
2022-03-01 10:42:07 -07:00
}
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
self.have_exclusive_lock = true;
Cache: fix unnecessary cache misses With the old logic, it was possible for a bunch of processes to queue up to update a cache entry, and then each to do so one at a time. Now, it rechecks whether there still a cache miss or another process has completed the work in the interim.
2023-05-11 00:51:41 -04:00
return true;
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// Obtain only the data needed to maintain a lock on the manifest file.
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
/// The `Manifest` remains safe to deinit.
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// Don't forget to call `writeManifest` before this!
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn toOwnedLock(self: *Manifest) Lock {
stage2: Cache: fix resource management of the deadlock debug code
2020-12-25 19:02:15 -07:00
const lock: Lock = .{
.manifest_file = self.manifest_file.?,
stage2: Cache: add debug deadlock detection code
2020-12-25 18:38:49 -07:00
};
cache: Fix LockViolation during C compilation paths (#13591) - C compilation flows didn't hold an exclusive lock on the cache manifest file when writing to it in all cases - On windows, explicitly unlock the file lock before closing it
2022-12-06 23:15:54 -05:00
stage2: Cache: fix resource management of the deadlock debug code
2020-12-25 19:02:15 -07:00
self.manifest_file = null;
return lock;
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
}
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
/// Releases the manifest file and frees any memory the Manifest was using.
/// `Manifest.hit` must be called first.
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
/// Don't forget to call `writeManifest` before this!
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
pub fn deinit(self: *Manifest) void {
Remove non-null assertion in `CacheHash.release()` People using the API as intended would never trigger this assertion anyway, but if someone has a non standard use case, I see no reason to make the program panic.
2020-04-30 19:54:40 -06:00
if (self.manifest_file) |file| {
cache: Fix LockViolation during C compilation paths (#13591) - C compilation flows didn't hold an exclusive lock on the cache manifest file when writing to it in all cases - On windows, explicitly unlock the file lock before closing it
2022-12-06 23:15:54 -05:00
if (builtin.os.tag == .windows) {
// See Lock.release for why this is required on Windows
file.unlock();
}
Remove non-null assertion in `CacheHash.release()` People using the API as intended would never trigger this assertion anyway, but if someone has a non standard use case, I see no reason to make the program panic.
2020-04-30 19:54:40 -06:00
file.close();
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
for (self.files.keys()) |*file| {
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
file.deinit(self.cache.gpa);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
self.files.deinit(self.cache.gpa);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
integrate Compile steps with file watching Updates the build runner to unconditionally require a zig lib directory parameter. This parameter is needed in order to correctly understand file system inputs from zig compiler subprocesses, since they will refer to "the zig lib directory", and the build runner needs to place file system watches on directories in there. The build runner's fanotify file watching implementation now accounts for when two or more Cache.Path instances compare unequal but ultimately refer to the same directory in the file system. Breaking change: std.Build no longer has a zig_lib_dir field. Instead, there is the Graph zig_lib_directory field, and individual Compile steps can still have their zig lib directories overridden. I think this is unlikely to break anyone's build in practice. The compiler now sends a "file_system_inputs" message to the build runner which shares the full set of files that were added to the cache system with the build system, so that the build runner can watch properly and redo the Compile step. This is implemented for whole cache mode but not yet for incremental cache mode.
2024-07-11 16:26:04 -07:00
pub fn populateFileSystemInputs(man: *Manifest, buf: *std.ArrayListUnmanaged(u8)) Allocator.Error!void {
std: update `std.builtin.Type` fields to follow naming conventions The compiler actually doesn't need any functional changes for this: Sema does reification based on the tag indices of `std.builtin.Type` already! So, no zig1.wasm update is necessary. This change is necessary to disallow name clashes between fields and decls on a type, which is a prerequisite of #9938.
2024-08-28 02:35:53 +01:00
assert(@typeInfo(std.zig.Server.Message.PathPrefix).@"enum".fields.len == man.cache.prefixes_len);
frontend: add file system inputs for incremental cache mode These are also used for whole cache mode in the case that any compile errors are emitted.
2024-07-11 18:28:05 -07:00
buf.clearRetainingCapacity();
integrate Compile steps with file watching Updates the build runner to unconditionally require a zig lib directory parameter. This parameter is needed in order to correctly understand file system inputs from zig compiler subprocesses, since they will refer to "the zig lib directory", and the build runner needs to place file system watches on directories in there. The build runner's fanotify file watching implementation now accounts for when two or more Cache.Path instances compare unequal but ultimately refer to the same directory in the file system. Breaking change: std.Build no longer has a zig_lib_dir field. Instead, there is the Graph zig_lib_directory field, and individual Compile steps can still have their zig lib directories overridden. I think this is unlikely to break anyone's build in practice. The compiler now sends a "file_system_inputs" message to the build runner which shares the full set of files that were added to the cache system with the build system, so that the build runner can watch properly and redo the Compile step. This is implemented for whole cache mode but not yet for incremental cache mode.
2024-07-11 16:26:04 -07:00
const gpa = man.cache.gpa;
const files = man.files.keys();
if (files.len > 0) {
for (files) |file| {
try buf.ensureUnusedCapacity(gpa, file.prefixed_path.sub_path.len + 2);
buf.appendAssumeCapacity(file.prefixed_path.prefix + 1);
buf.appendSliceAssumeCapacity(file.prefixed_path.sub_path);
buf.appendAssumeCapacity(0);
}
// The null byte is a separator, not a terminator.
buf.items.len -= 1;
}
}
add sub-compilation cache inputs to parents in whole mode closes #20782
2024-07-24 19:40:54 -07:00
pub fn populateOtherManifest(man: *Manifest, other: *Manifest, prefix_map: [4]u8) Allocator.Error!void {
const gpa = other.cache.gpa;
std: update `std.builtin.Type` fields to follow naming conventions The compiler actually doesn't need any functional changes for this: Sema does reification based on the tag indices of `std.builtin.Type` already! So, no zig1.wasm update is necessary. This change is necessary to disallow name clashes between fields and decls on a type, which is a prerequisite of #9938.
2024-08-28 02:35:53 +01:00
assert(@typeInfo(std.zig.Server.Message.PathPrefix).@"enum".fields.len == man.cache.prefixes_len);
add sub-compilation cache inputs to parents in whole mode closes #20782
2024-07-24 19:40:54 -07:00
assert(man.cache.prefixes_len == 4);
for (man.files.keys()) |file| {
const prefixed_path: PrefixedPath = .{
.prefix = prefix_map[file.prefixed_path.prefix],
.sub_path = try gpa.dupe(u8, file.prefixed_path.sub_path),
};
errdefer gpa.free(prefixed_path.sub_path);
const gop = try other.files.getOrPutAdapted(gpa, prefixed_path, FilesAdapter{});
errdefer _ = other.files.pop();
if (gop.found_existing) {
gpa.free(prefixed_path.sub_path);
continue;
}
gop.key_ptr.* = .{
.prefixed_path = prefixed_path,
.max_file_size = file.max_file_size,
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
.handle = file.handle,
add sub-compilation cache inputs to parents in whole mode closes #20782
2024-07-24 19:40:54 -07:00
.stat = file.stat,
.bin_digest = file.bin_digest,
.contents = null,
};
other.hash.hasher.update(&gop.key_ptr.bin_digest);
}
}
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
};
fixups to previous commit * std.fs.Dir.readFile: add doc comments to explain what it means when the returned slice has the same length as the supplied buffer. * introduce readSmallFile / writeSmallFile to abstract over the decision to use symlink or file contents to store data.
2020-10-09 16:45:39 -07:00
/// On operating systems that support symlinks, does a readlink. On other operating systems,
/// uses the file contents. Windows supports symlinks but only with elevated privileges, so
/// it is treated as not supporting symlinks.
pub fn readSmallFile(dir: fs.Dir, sub_path: []const u8, buffer: []u8) ![]u8 {
migrate from `std.Target.current` to `@import("builtin").target` closes #9388 closes #9321
2021-10-04 23:47:27 -07:00
if (builtin.os.tag == .windows) {
fixups to previous commit * std.fs.Dir.readFile: add doc comments to explain what it means when the returned slice has the same length as the supplied buffer. * introduce readSmallFile / writeSmallFile to abstract over the decision to use symlink or file contents to store data.
2020-10-09 16:45:39 -07:00
return dir.readFile(sub_path, buffer);
} else {
return dir.readLink(sub_path, buffer);
}
}
/// On operating systems that support symlinks, does a symlink. On other operating systems,
/// uses the file contents. Windows supports symlinks but only with elevated privileges, so
/// it is treated as not supporting symlinks.
/// `data` must be a valid UTF-8 encoded file path and 255 bytes or fewer.
pub fn writeSmallFile(dir: fs.Dir, sub_path: []const u8, data: []const u8) !void {
assert(data.len <= 255);
migrate from `std.Target.current` to `@import("builtin").target` closes #9388 closes #9321
2021-10-04 23:47:27 -07:00
if (builtin.os.tag == .windows) {
Rename Dir.writeFile2 -> Dir.writeFile and update all callsites writeFile was deprecated in favor of writeFile2 in f645022d16361865e24582d28f1e62312fbc73bb. This commit renames writeFile2 to writeFile and makes writeFile2 a compile error.
2024-05-02 20:54:48 -07:00
return dir.writeFile(.{ .sub_path = sub_path, .data = data });
fixups to previous commit * std.fs.Dir.readFile: add doc comments to explain what it means when the returned slice has the same length as the supplied buffer. * introduce readSmallFile / writeSmallFile to abstract over the decision to use symlink or file contents to store data.
2020-10-09 16:45:39 -07:00
} else {
return dir.symLink(data, sub_path, .{});
}
}
std.Build.Cache.hit: more discipline in error handling Previous commits 2b0929929d67e222ca6a9523a3a594ed456c4a51 4ea2f441df36cec61e1017f4d795d4037326c98c had this text: > There are no dir components, so you would think that this was > unreachable, however we have observed on macOS two processes racing to > do openat() with O_CREAT manifest in ENOENT. This appears to have been a misunderstanding based on the issue report #12138 and corresponding PR #12139 in which the steps to reproduce removed the cache directory in a loop which also executed detached Zig compiler processes. There is no evidence for the macOS kernel bug however the ENOENT is easily explained by the removal of the cache directory. This commit reverts those commits, ultimately reporting the ENOENT as an error rather than repeating the create file operation. However this commit also adds an explicit error set to `std.Build.Cache.hit` as well as changing the `failed_file_index` to a proper diagnostic field that fully communicates what failed, leading to more informative error messages on failure to check the cache. The equivalent failure when occuring for AstGen performs a fatal process kill, reasoning being that the compiler has an invariant of the cache directory not being yanked out from underneath it while executing. This could be made a more granular error in the future but I suspect such thing is not valuable to pursue. Related to #18340 but does not solve it.
2024-12-10 17:43:42 -08:00
fn hashFile(file: fs.File, bin_digest: *[Hasher.mac_length]u8) fs.File.PReadError!void {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
var buf: [1024]u8 = undefined;
cache_hash: hash function change This makes the `cache_hash` hash function easier to replace. BLAKE3 would be a natural fit for hashing large files, but: - second preimage resistance is not necessary for the cache_hash use cases - our BLAKE3 implementation is currently very slow Switch to SipHash128, which gives us an immediate speed boost.
2020-08-21 15:08:15 +02:00
var hasher = hasher_init;
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
var off: u64 = 0;
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
while (true) {
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
const bytes_read = try file.pread(&buf, off);
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
if (bytes_read == 0) break;
cache_hash: hash function change This makes the `cache_hash` hash function easier to replace. BLAKE3 would be a natural fit for hashing large files, but: - second preimage resistance is not necessary for the cache_hash use cases - our BLAKE3 implementation is currently very slow Switch to SipHash128, which gives us an immediate speed boost.
2020-08-21 15:08:15 +02:00
hasher.update(buf[0..bytes_read]);
rework linker inputs * Compilation.objects changes to Compilation.link_inputs which stores objects, archives, windows resources, shared objects, and strings intended to be put directly into the dynamic section. Order is now preserved between all of these kinds of linker inputs. If it is determined the order does not matter for a particular kind of linker input, that item should be moved to a different array. * rename system_libs to windows_libs * untangle library lookup from CLI types * when doing library lookup, instead of using access syscalls, go ahead and open the files and keep the handles around for passing to the cache system and the linker. * during library lookup and cache file hashing, use positioned reads to avoid affecting the file seek position. * library directories are opened in the CLI and converted to Directory objects, warnings emitted for those that cannot be opened.
2024-10-16 12:14:19 -07:00
off += bytes_read;
Add max_file_size argument
2020-05-01 23:06:10 -06:00
}
cache_hash: hash function change This makes the `cache_hash` hash function easier to replace. BLAKE3 would be a natural fit for hashing large files, but: - second preimage resistance is not necessary for the cache_hash use cases - our BLAKE3 implementation is currently very slow Switch to SipHash128, which gives us an immediate speed boost.
2020-08-21 15:08:15 +02:00
hasher.final(bin_digest);
Partially implement cache hash API in zig
2020-03-05 00:07:17 -07:00
}
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
// Create/Write a file, close it, then grab its stat.mtime timestamp.
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
fn testGetCurrentFileTimestamp(dir: fs.Dir) !i128 {
std.Build.Cache: remove 'test-filetimestamp.tmp' once timestamp returned
2023-03-03 11:04:08 +03:30
const test_out_file = "test-filetimestamp.tmp";
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
var file = try dir.createFile(test_out_file, .{
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
.read = true,
Cache: improvements to previous commit * put `recent_problematic_timestamp` onto `Cache` so that it can be shared by multiple Manifest instances. * make `isProblematicTimestamp` return true on any filesystem error. * save 1 syscall by using truncate=true in createFile instead of calling `setEndPos`.
2021-12-09 18:55:20 -07:00
.truncate = true,
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
});
std.Build.Cache: remove 'test-filetimestamp.tmp' once timestamp returned
2023-03-03 11:04:08 +03:30
defer {
file.close();
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
dir.deleteFile(test_out_file) catch {};
std.Build.Cache: remove 'test-filetimestamp.tmp' once timestamp returned
2023-03-03 11:04:08 +03:30
}
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
Cache: improvements to previous commit * put `recent_problematic_timestamp` onto `Cache` so that it can be shared by multiple Manifest instances. * make `isProblematicTimestamp` return true on any filesystem error. * save 1 syscall by using truncate=true in createFile instead of calling `setEndPos`.
2021-12-09 18:55:20 -07:00
return (try file.stat()).mtime;
Check for problematic timestamps
2020-04-11 16:01:17 -06:00
}
Remove unnecessary contents field from File It was causing a segfault on `mipsel` architecture, not sure why other architectures weren't affected.
2020-04-07 23:57:59 -06:00
test "cache file and then recall it" {
migrate from `std.Target.current` to `@import("builtin").target` closes #9388 closes #9321
2021-10-04 23:47:27 -07:00
if (builtin.os.tag == .wasi) {
fix std lib tests for WASI
2020-05-25 19:46:28 -04:00
// https://github.com/ziglang/zig/issues/5437
return error.SkipZigTest;
}
stage2: Cache: fix resource management of the deadlock debug code
2020-12-25 19:02:15 -07:00
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
Remove up files created in test at end of test
2020-03-08 15:13:40 -06:00
const temp_file = "test.txt";
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
const temp_manifest_dir = "temp_manifest_dir";
Rename Dir.writeFile2 -> Dir.writeFile and update all callsites writeFile was deprecated in favor of writeFile2 in f645022d16361865e24582d28f1e62312fbc73bb. This commit renames writeFile2 to writeFile and makes writeFile2 a compile error.
2024-05-02 20:54:48 -07:00
try tmp.dir.writeFile(.{ .sub_path = temp_file, .data = "Hello, world!\n" });
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
// Wait for file timestamps to tick
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
const initial_time = try testGetCurrentFileTimestamp(tmp.dir);
while ((try testGetCurrentFileTimestamp(tmp.dir)) == initial_time) {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
std.time.sleep(1);
}
std.Build.Cache: add HexDigest type
2023-12-11 23:08:03 +01:00
var digest1: HexDigest = undefined;
var digest2: HexDigest = undefined;
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
{
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
var cache = Cache{
.gpa = testing.allocator,
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
.manifest_dir = try tmp.dir.makeOpenPath(temp_manifest_dir, .{}),
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
};
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
cache.addPrefix(.{ .path = null, .handle = tmp.dir });
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
defer cache.manifest_dir.close();
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
{
var ch = cache.obtain();
defer ch.deinit();
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
ch.hash.add(true);
ch.hash.add(@as(u16, 1234));
ch.hash.addBytes("1234");
_ = try ch.addFile(temp_file, null);
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// There should be nothing in the cache
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expectEqual(false, try ch.hit());
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
digest1 = ch.final();
try ch.writeManifest();
}
{
var ch = cache.obtain();
defer ch.deinit();
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
ch.hash.add(true);
ch.hash.add(@as(u16, 1234));
ch.hash.addBytes("1234");
_ = try ch.addFile(temp_file, null);
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// Cache hit! We just "built" the same file
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expect(try ch.hit());
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
digest2 = ch.final();
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
Fix another LockViolation case on Windows (#14162) - Add an assert that an exclusive lock is help to writeManifest - Only call writeManifest in updateCObject if an exclusive lock is held - cache: fixup test to verify hits don't take an exclusive lock, instead of writing the manifest
2023-01-04 14:51:43 -05:00
try testing.expectEqual(false, ch.have_exclusive_lock);
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
}
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expectEqual(digest1, digest2);
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
}
Fix memory leak in cache_hash
2020-03-05 22:59:19 -07:00
}
Check for problematic timestamps
2020-04-11 16:01:17 -06:00
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
test "check that changing a file makes cache fail" {
migrate from `std.Target.current` to `@import("builtin").target` closes #9388 closes #9321
2021-10-04 23:47:27 -07:00
if (builtin.os.tag == .wasi) {
fix std lib tests for WASI
2020-05-25 19:46:28 -04:00
// https://github.com/ziglang/zig/issues/5437
return error.SkipZigTest;
}
Revert "Revert "Merge pull request #17637 from jacobly0/x86_64-test-std"" This reverts commit 6f0198cadbe29294f2bf3153a27beebd64377566.
2023-10-22 15:46:33 -04:00
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
const temp_file = "cache_hash_change_file_test.txt";
const temp_manifest_dir = "cache_hash_change_file_manifest_dir";
Add max_file_size argument
2020-05-01 23:06:10 -06:00
const original_temp_file_contents = "Hello, world!\n";
const updated_temp_file_contents = "Hello, world; but updated!\n";
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
Rename Dir.writeFile2 -> Dir.writeFile and update all callsites writeFile was deprecated in favor of writeFile2 in f645022d16361865e24582d28f1e62312fbc73bb. This commit renames writeFile2 to writeFile and makes writeFile2 a compile error.
2024-05-02 20:54:48 -07:00
try tmp.dir.writeFile(.{ .sub_path = temp_file, .data = original_temp_file_contents });
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
// Wait for file timestamps to tick
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
const initial_time = try testGetCurrentFileTimestamp(tmp.dir);
while ((try testGetCurrentFileTimestamp(tmp.dir)) == initial_time) {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
std.time.sleep(1);
}
std.Build.Cache: add HexDigest type
2023-12-11 23:08:03 +01:00
var digest1: HexDigest = undefined;
var digest2: HexDigest = undefined;
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
{
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
var cache = Cache{
.gpa = testing.allocator,
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
.manifest_dir = try tmp.dir.makeOpenPath(temp_manifest_dir, .{}),
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
};
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
cache.addPrefix(.{ .path = null, .handle = tmp.dir });
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
defer cache.manifest_dir.close();
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
{
var ch = cache.obtain();
defer ch.deinit();
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
ch.hash.addBytes("1234");
const temp_file_idx = try ch.addFile(temp_file, 100);
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// There should be nothing in the cache
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expectEqual(false, try ch.hit());
Add max_file_size argument
2020-05-01 23:06:10 -06:00
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
try testing.expect(mem.eql(u8, original_temp_file_contents, ch.files.keys()[temp_file_idx].contents.?));
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
digest1 = ch.final();
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
try ch.writeManifest();
}
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
Rename Dir.writeFile2 -> Dir.writeFile and update all callsites writeFile was deprecated in favor of writeFile2 in f645022d16361865e24582d28f1e62312fbc73bb. This commit renames writeFile2 to writeFile and makes writeFile2 a compile error.
2024-05-02 20:54:48 -07:00
try tmp.dir.writeFile(.{ .sub_path = temp_file, .data = updated_temp_file_contents });
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
{
var ch = cache.obtain();
defer ch.deinit();
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
ch.hash.addBytes("1234");
const temp_file_idx = try ch.addFile(temp_file, 100);
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// A file that we depend on has been updated, so the cache should not contain an entry for it
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expectEqual(false, try ch.hit());
Add max_file_size argument
2020-05-01 23:06:10 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// The cache system does not keep the contents of re-hashed input files.
std.Build.Cache: use an array hash map for files Rather than an ArrayList. Provides deduplication.
2024-03-21 19:53:24 -07:00
try testing.expect(ch.files.keys()[temp_file_idx].contents == null);
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
digest2 = ch.final();
try ch.writeManifest();
}
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expect(!mem.eql(u8, digest1[0..], digest2[0..]));
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
}
Add test checking file changes invalidate cache
2020-04-14 21:37:35 -06:00
}
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
test "no file inputs" {
migrate from `std.Target.current` to `@import("builtin").target` closes #9388 closes #9321
2021-10-04 23:47:27 -07:00
if (builtin.os.tag == .wasi) {
fix std lib tests for WASI
2020-05-25 19:46:28 -04:00
// https://github.com/ziglang/zig/issues/5437
return error.SkipZigTest;
}
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
const temp_manifest_dir = "no_file_inputs_manifest_dir";
std.Build.Cache: add HexDigest type
2023-12-11 23:08:03 +01:00
var digest1: HexDigest = undefined;
var digest2: HexDigest = undefined;
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
var cache = Cache{
.gpa = testing.allocator,
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
.manifest_dir = try tmp.dir.makeOpenPath(temp_manifest_dir, .{}),
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
};
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
cache.addPrefix(.{ .path = null, .handle = tmp.dir });
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
defer cache.manifest_dir.close();
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
{
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
var man = cache.obtain();
defer man.deinit();
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
man.hash.addBytes("1234");
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
// There should be nothing in the cache
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
try testing.expectEqual(false, try man.hit());
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
digest1 = man.final();
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
try man.writeManifest();
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
}
{
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
var man = cache.obtain();
defer man.deinit();
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
man.hash.addBytes("1234");
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
stage2: Cache system handles shared objects Fixes #9139 Fixes #9187
2021-06-27 22:33:17 -07:00
try testing.expect(try man.hit());
digest2 = man.final();
Fix another LockViolation case on Windows (#14162) - Add an assert that an exclusive lock is help to writeManifest - Only call writeManifest in updateCObject if an exclusive lock is held - cache: fixup test to verify hits don't take an exclusive lock, instead of writing the manifest
2023-01-04 14:51:43 -05:00
try testing.expectEqual(false, man.have_exclusive_lock);
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
}
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expectEqual(digest1, digest2);
Add "no file inputs" test It checks whether the cache will respond correctly to inputs that don't initially depend on filesystem state. In that case, we have to check for the existence of a manifest file, instead of relying on reading the list of entries to tell us if the cache is invalid.
2020-04-14 21:39:34 -06:00
}
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
stage2: implement @cImport Also rename Cache.CacheHash to Cache.Manifest
2020-09-24 16:22:45 -07:00
test "Manifest with files added after initial hash work" {
migrate from `std.Target.current` to `@import("builtin").target` closes #9388 closes #9321
2021-10-04 23:47:27 -07:00
if (builtin.os.tag == .wasi) {
fix std lib tests for WASI
2020-05-25 19:46:28 -04:00
// https://github.com/ziglang/zig/issues/5437
return error.SkipZigTest;
}
Revert "Revert "Merge pull request #17637 from jacobly0/x86_64-test-std"" This reverts commit 6f0198cadbe29294f2bf3153a27beebd64377566.
2023-10-22 15:46:33 -04:00
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
var tmp = testing.tmpDir(.{});
defer tmp.cleanup();
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
const temp_file1 = "cache_hash_post_file_test1.txt";
const temp_file2 = "cache_hash_post_file_test2.txt";
const temp_manifest_dir = "cache_hash_post_file_manifest_dir";
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
Rename Dir.writeFile2 -> Dir.writeFile and update all callsites writeFile was deprecated in favor of writeFile2 in f645022d16361865e24582d28f1e62312fbc73bb. This commit renames writeFile2 to writeFile and makes writeFile2 a compile error.
2024-05-02 20:54:48 -07:00
try tmp.dir.writeFile(.{ .sub_path = temp_file1, .data = "Hello, world!\n" });
try tmp.dir.writeFile(.{ .sub_path = temp_file2, .data = "Hello world the second!\n" });
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
// Wait for file timestamps to tick
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
const initial_time = try testGetCurrentFileTimestamp(tmp.dir);
while ((try testGetCurrentFileTimestamp(tmp.dir)) == initial_time) {
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
std.time.sleep(1);
}
std.Build.Cache: add HexDigest type
2023-12-11 23:08:03 +01:00
var digest1: HexDigest = undefined;
var digest2: HexDigest = undefined;
var digest3: HexDigest = undefined;
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
{
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
var cache = Cache{
.gpa = testing.allocator,
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
.manifest_dir = try tmp.dir.makeOpenPath(temp_manifest_dir, .{}),
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
};
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
cache.addPrefix(.{ .path = null, .handle = tmp.dir });
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
defer cache.manifest_dir.close();
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
{
var ch = cache.obtain();
defer ch.deinit();
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
ch.hash.addBytes("1234");
_ = try ch.addFile(temp_file1, null);
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// There should be nothing in the cache
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expectEqual(false, try ch.hit());
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
_ = try ch.addFilePost(temp_file2);
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
digest1 = ch.final();
try ch.writeManifest();
}
{
var ch = cache.obtain();
defer ch.deinit();
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
ch.hash.addBytes("1234");
_ = try ch.addFile(temp_file1, null);
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expect(try ch.hit());
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
digest2 = ch.final();
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
Fix another LockViolation case on Windows (#14162) - Add an assert that an exclusive lock is help to writeManifest - Only call writeManifest in updateCObject if an exclusive lock is held - cache: fixup test to verify hits don't take an exclusive lock, instead of writing the manifest
2023-01-04 14:51:43 -05:00
try testing.expectEqual(false, ch.have_exclusive_lock);
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
}
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expect(mem.eql(u8, &digest1, &digest2));
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// Modify the file added after initial hash
Rename Dir.writeFile2 -> Dir.writeFile and update all callsites writeFile was deprecated in favor of writeFile2 in f645022d16361865e24582d28f1e62312fbc73bb. This commit renames writeFile2 to writeFile and makes writeFile2 a compile error.
2024-05-02 20:54:48 -07:00
try tmp.dir.writeFile(.{ .sub_path = temp_file2, .data = "Hello world the second, updated\n" });
improvements to self-hosted cache hash system * change miscellaneous things to more idiomatic zig style * change the digest length to 24 bytes instead of 48. This is still 70 more bits than UUIDs. For an analysis of probability of collisions, see: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions * fix the API having the possibility of mismatched allocators * fix some error paths to behave properly * modify the guarantees about when file contents are loaded for input files * pwrite instead of seek + write * implement isProblematicTimestamp * fix tests with regards to a working isProblematicTimestamp function. this requires sleeping until the current timestamp becomes unproblematic. * introduce std.fs.File.INode, a cross platform type abstraction so that cache hash implementation does not need to reach into std.os.
2020-05-25 19:29:03 -04:00
Cache: fix two issues with isProblematicTimestamp 1. It was looking for trailing zero bits when it should be looking for trailing decimal zeros. 2. Clock timestamps had more precision than the actual file timestamps The fix is to grab a timestamp from a 'just now changed' temp file. This timestamp is "problematic". Any file timestamp greater than or equal to this timestamp is considered problematic. File timestamps **prior** to this **can** be trusted. Downside is that it causes a disk I/O to write to and then read the timestamp from this file ~1ms on my system. This is partially mitigated by keeping track of the most recent problematic timestamp, and only checking for a new problematic timestamp when checking a timestamp that is equal to or larger than the last problematic one. This fixes #6082.
2021-10-10 21:57:26 -07:00
// Wait for file timestamps to tick
std.Build.Cache: make unit tests not depend on cwd This makes them more resilient to being run multiple times by multiple different processes at the same time.
2023-03-14 16:38:14 -07:00
const initial_time2 = try testGetCurrentFileTimestamp(tmp.dir);
while ((try testGetCurrentFileTimestamp(tmp.dir)) == initial_time2) {
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
std.time.sleep(1);
}
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
{
var ch = cache.obtain();
defer ch.deinit();
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
ch.hash.addBytes("1234");
_ = try ch.addFile(temp_file1, null);
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
// A file that we depend on has been updated, so the cache should not contain an entry for it
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expectEqual(false, try ch.hit());
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
_ = try ch.addFilePost(temp_file2);
std.cache_hash: break up the API and improve implementation into smaller exposed components and expose all of them. This makes it more flexible. `*const Cache` is now passed in with an open manifest dir handle which the caller is responsible for managing. Expose some of the base64 stuff. Extract the hash helper functions into `HashHelper` and add some more methods such as addOptional and addListOfFiles. Add `CacheHash.toOwnedLock` so that you can deinitialize everything except the open file handle which represents the file system lock on the build artifacts. Use ArrayListUnmanaged, saving space per allocated CacheHash. Avoid 1 memory allocation in hit() with a static buffer. hit() returns a bool; caller code is responsible for calling final() in either case. This is a simpler and easier to use API. writeManifest() is no longer called from deinit() with errors ignored.
2020-09-13 18:04:17 -07:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
digest3 = ch.final();
Change null pointer test to `addFilePost` test
2020-04-30 19:47:04 -06:00
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
try ch.writeManifest();
}
update usage of std.testing in stage2
2021-05-05 21:29:16 +03:00
try testing.expect(!mem.eql(u8, &digest1, &digest3));
stage2: building glibc shared objects * caching system: use 16 bytes siphash final(), there was a bug in the std lib that wasn't catching undefined values for 18 bytes. fixed in master branch. * fix caching system unit test logic to not cause error.TextBusy on windows * port the logic from stage1 for building glibc shared objects * add is_native_os to the base cache hash * fix incorrectly freeing crt_files key (which is always a reference to global static constant data) * fix 2 use-after-free in loading glibc metadata * fix memory leak in buildCRTFile (errdefer instead of defer on arena)
2020-09-16 03:02:46 -07:00
}
Add test case for fix in previous commit
2020-04-30 17:06:03 -06:00
}
Reference in New Issue Copy Permalink