[Performance] Optimize strided transpose and fp8 1x128 quantization (#79743)
* [Performance] Optimize strided transpose and fp8 1x128 quantization Five changes on one path. They share a trigger: sharding stage1 with split_param and fused optimizer states gives every fused parameter but the first a non-zero offset, and that offset used to disqualify the fast paths below. paddle/phi/kernels/stride/matmul_stride_kernel.cu is_only_transposed_tensor() rejected a non-zero offset. Offset is orthogonal to the question it asks: only dims and strides are reordered, the caller writes the original offset back, data() adds it on every access, and is_contiguous() never looks at it. With the rejection gone, w.t() on a fused parameter folds into the cuBLAS transpose flag instead of materializing a contiguous copy with a non-coalesced read. paddle/phi/kernels/gpu/contiguous_kernel.cu The sibling check in is_only_transposed() had the same rejection, which sent .contiguous() on a transposed view to ContiguousCaseOneFunc, one element per thread with no coalescing. Dropping the test requires the accompanying set_meta() fix: out is freshly allocated and contiguous, so its offset must be zero rather than inherited from the input. The removed test was the only thing keeping that latent bug unreachable. paddle/phi/kernels/funcs/transpose_function.cuh TilingSwapDim1And2Vec, a bit-exact equivalent of TilingSwapDim1And2 for 2-byte dtypes. Tile coordinates come from a 3D grid instead of five runtime integer divisions per thread, tiles are larger, and the global accesses are vectorized; shared memory is still accessed element by element so the pad can stay odd and avoid bank conflicts. It runs only when both transposed extents are even, both base pointers are 4-byte aligned, the grid fits in 65535, and there are enough 64x64 tiles to fill the device. Every other case keeps the existing scalar tiling. PermuteDispatch also clamps GetVecSize(). It can return 8 for 2-byte dtypes and neither switch has a case for 8, so no kernel would be launched and the output would be left uninitialized. This one is defensive: no default configuration that reaches 8 was found. paddle/phi/kernels/legacy/gpu/fp8_quant_blockwise_kernel.cu quantize_1x128_kernel_v128, a 128-bit rewrite of quantize_1x128_kernel covering the non-transposing instantiation. It is selected only when the input and output base pointers are both 16-byte aligned; otherwise the scalar kernel runs. The only reordering is inside the abs-max reduction, and the max used there suppresses NaN and is associative, so the result is unchanged. Tests test/legacy_test/test_stride_offset_kernel.py walks both sides of every selection gate of the new transpose kernel, materializes views whose offset is non-zero, and checks that a folded matmul does not depend on the offset. Transpose is pure data movement, so the reference is an exact bit comparison against numpy rather than a tolerance. test/legacy_test/test_fp8_quant.py gains base-pointer alignment coverage, a cross-check of the new kernel against the scalar one, and abs-max edge cases built so that a changed reduction order would show: NaN, infinities, zero blocks, and single outliers placed at a different position in each block. * Fix pre-sm_80 build of quantize_1x128_kernel_v128 The packed bf16 amax used __habs2 / __hmax2 / __low2bfloat16 / __high2bfloat16 directly. On older toolkits those are declared only under __CUDA_ARCH__ >= 800, and because the names do not depend on the template parameter they are diagnosed even in the discarded branch of the if-constexpr, so the float16 instantiation failed to compile for sm_70. Use the file's own device_abs / device_max helpers, which already carry the arch guard with a float fallback. The kernel is memory bound, so halving the number of max operations was not where its speedup came from, and max stays associative so the result is unchanged. * Pick the matmul fold test dtype from the device compute capability cuBLAS rejects a plain bfloat16 gemm below compute capability 80, so the hardcoded bfloat16 operands made the offset-fold tests fail on pre-sm_80 CI machines. float16 is also two bytes wide, so the offset alignment the test relies on is unchanged. * Gate the float8 tile transpose on the base pointer alignment The kernel moves eight bytes at a time but the gate only looked at the extents, so a float8 view starting at an offset that is not a multiple of eight faulted with a misaligned address. A one-byte dtype makes this reachable from any slice, and now also from ContiguousKernel, which no longer refuses a transposed view with a non-zero offset. * Join the float8 reference line so ruff-format leaves it alone
S
SUN Dong committed
3f83b3ab44fa2e252dd436d341b179b23d7823b0
Parent: 71b2f32
Committed by GitHub <noreply@github.com>
on 9/10/2026, 7:05:09 AM