SIGN IN SIGN UP

[Performance] Optimize strided transpose and fp8 1x128 quantization (#79743)

* [Performance] Optimize strided transpose and fp8 1x128 quantization

Five changes on one path. They share a trigger: sharding stage1 with
split_param and fused optimizer states gives every fused parameter but
the first a non-zero offset, and that offset used to disqualify the fast
paths below.

paddle/phi/kernels/stride/matmul_stride_kernel.cu
  is_only_transposed_tensor() rejected a non-zero offset. Offset is
  orthogonal to the question it asks: only dims and strides are
  reordered, the caller writes the original offset back, data() adds it
  on every access, and is_contiguous() never looks at it. With the
  rejection gone, w.t() on a fused parameter folds into the cuBLAS
  transpose flag instead of materializing a contiguous copy with a
  non-coalesced read.

paddle/phi/kernels/gpu/contiguous_kernel.cu
  The sibling check in is_only_transposed() had the same rejection,
  which sent .contiguous() on a transposed view to
  ContiguousCaseOneFunc, one element per thread with no coalescing.
  Dropping the test requires the accompanying set_meta() fix: out is
  freshly allocated and contiguous, so its offset must be zero rather
  than inherited from the input. The removed test was the only thing
  keeping that latent bug unreachable.

paddle/phi/kernels/funcs/transpose_function.cuh
  TilingSwapDim1And2Vec, a bit-exact equivalent of TilingSwapDim1And2
  for 2-byte dtypes. Tile coordinates come from a 3D grid instead of
  five runtime integer divisions per thread, tiles are larger, and the
  global accesses are vectorized; shared memory is still accessed
  element by element so the pad can stay odd and avoid bank conflicts.
  It runs only when both transposed extents are even, both base
  pointers are 4-byte aligned, the grid fits in 65535, and there are
  enough 64x64 tiles to fill the device. Every other case keeps the
  existing scalar tiling.

  PermuteDispatch also clamps GetVecSize(). It can return 8 for 2-byte
  dtypes and neither switch has a case for 8, so no kernel would be
  launched and the output would be left uninitialized. This one is
  defensive: no default configuration that reaches 8 was found.

paddle/phi/kernels/legacy/gpu/fp8_quant_blockwise_kernel.cu
  quantize_1x128_kernel_v128, a 128-bit rewrite of
  quantize_1x128_kernel covering the non-transposing instantiation. It
  is selected only when the input and output base pointers are both
  16-byte aligned; otherwise the scalar kernel runs. The only
  reordering is inside the abs-max reduction, and the max used there
  suppresses NaN and is associative, so the result is unchanged.

Tests
  test/legacy_test/test_stride_offset_kernel.py walks both sides of
  every selection gate of the new transpose kernel, materializes views
  whose offset is non-zero, and checks that a folded matmul does not
  depend on the offset. Transpose is pure data movement, so the
  reference is an exact bit comparison against numpy rather than a
  tolerance.

  test/legacy_test/test_fp8_quant.py gains base-pointer alignment
  coverage, a cross-check of the new kernel against the scalar one, and
  abs-max edge cases built so that a changed reduction order would
  show: NaN, infinities, zero blocks, and single outliers placed at a
  different position in each block.

* Fix pre-sm_80 build of quantize_1x128_kernel_v128

The packed bf16 amax used __habs2 / __hmax2 / __low2bfloat16 /
__high2bfloat16 directly. On older toolkits those are declared only under
__CUDA_ARCH__ >= 800, and because the names do not depend on the template
parameter they are diagnosed even in the discarded branch of the
if-constexpr, so the float16 instantiation failed to compile for sm_70.

Use the file's own device_abs / device_max helpers, which already carry the
arch guard with a float fallback. The kernel is memory bound, so halving the
number of max operations was not where its speedup came from, and max stays
associative so the result is unchanged.

* Pick the matmul fold test dtype from the device compute capability

cuBLAS rejects a plain bfloat16 gemm below compute capability 80, so the
hardcoded bfloat16 operands made the offset-fold tests fail on pre-sm_80
CI machines. float16 is also two bytes wide, so the offset alignment the
test relies on is unchanged.

* Gate the float8 tile transpose on the base pointer alignment

The kernel moves eight bytes at a time but the gate only looked at the
extents, so a float8 view starting at an offset that is not a multiple of
eight faulted with a misaligned address. A one-byte dtype makes this
reachable from any slice, and now also from ContiguousKernel, which no
longer refuses a transposed view with a non-zero offset.

* Join the float8 reference line so ruff-format leaves it alone
S
SUN Dong committed
3f83b3ab44fa2e252dd436d341b179b23d7823b0
Parent: 71b2f32
Committed by GitHub <noreply@github.com> on 9/10/2026, 7:05:09 AM