Fix python gpu build (#3081)

The problem was introduced in #2999. After that, we can't publish our python wheels to pypi because the package size is too big.