It would be nice to be able to use the set_index method to sort the dataframe by a particular column.

There are currently two implementations for this, one in dask.dataframe and one in dask-cudf which uses a batcher sorting net. While most dask-cudf code has been removed in favor of the dask.dataframe implementations this sorting code has remained, mostly because I don't understand it fully, and don't know if there was a reason for this particular implementation.

Why was this implementation chosen? Was this discussed somewhere? Alternatively @sklam, do you have any information here?

cc @kkraus14 @randerzander

Update cudf.to_parquet to use new GPU accelerated Parquet writer. This including creating the appropriate c++ interface in io_writers and io_functions along with modifications to parquet pyx and pxd files.

This closes #3574

In a conversation with @kkraus14 about cudf usage with cuml+gridsearch we looked at cudf performance. Attached is profile plot of running gridsearch+cuml+cudf.

Folks can download the full dask profile here: https://gist.github.com/quasiben/1da49c5aa6e61d979dd42ce6c50e79b3

In the image above you can see that the computation is spending ~80% in the iloc call. My initial thought was that iloc/_prepare_series_for_add could be improved. I believe @kkraus14 suggested we look at host_to_device transfers see if we can build requisite indicies in cuda/cupy/numba_cuda instead of numpy (this is required during splitting/kfold calls)

Creating and interfacing with the gdf_column C struct is rife with problems. We need better abstraction(s) to make libcudf a safer and more pleasant library to use and develop.

In lieu of adding support for any new column types, the initial goal of a cudf::column design is to ease the current pain points in creating and interfacing with the column data structure in libcudf.

Goals:

• Identify pain points with existing gdf_column structure
• Derive requirements for an abstraction or set of abstractions to ease those pain points
• Define an API design that satisfies the requirements
• Provide a working implementation of the design

Non-Goals

• Derive requirements to support new column types, e.g., variable width elements, compressed columns, etc.
• Support delayed materialization or lazy evaluation

Note that a “Non-Goal” is not something that we want to expressly forbid in our redesign, but rather are not the focus of the current effort. Whenever possible, we can make design decisions that will enable these “Non-Goals” sometime in the future, so long as those decisions do not compromise the timely accomplishment of the above “Goals”

Process

1. Gather pain points

• Those who wish to participate should list 3-5 pain points (in priority order) that they would like to solve with the column redesign.
  • Note that choosing to participate implies a commitment to putting in the effort to derive requirements and provide feedback on designs, i.e., if you want something to change, you’re expected to put in the work to make it happen.
• Pain points should be submitted by responding to this issue.
• @jrhemstad will take responsibility for gathering pain points and distilling/organizing based on functional area.
• Proposed Deadline: 0.7 release

2. Derive requirements

• Distill pain points into satisfiable requirements
• @jrhemstad will take responsibility for providing an initial draft of requirements from pain points and distributing for feedback.
• Stakeholders will provide feedback on requirements and iterate until consensus is reached on initial requirements
• Proposed Deadline: 0.8 Release

3. Design Mock Up

• Create draft interface of class(es) that attempt to satisfy requirements.
• APIs should be fully Doxymented.
• Code does not need to function nor compile
• Design should be submitted via a PR to cuDF
• TBD will take responsibility for providing an initial interface design
• Stakeholders will provide feedback and iterate until consensus is reached on design
• Proposed Deadline: 0.8 Release

4. Implementation

• Implement the agreed upon interface
• Should provide Google Test unit tests
• Implementation/testing will likely expose necessary design changes
• Implementation should be submitted as a PR to cuDF
• TBD will take responsibility for implementing/testing the design
• Stakeholders will review implementation PR until consensus is reached
• Proposed Deadline 0.8 Release

5. Initial Refactor

• Two candidate libcudf features shall be chosen for refactoring to use the new cudf::column abstraction
• Two developers (TBD) will take responsibility for refactoring the features (one each) to use the newly designed abstraction(s) and submitting a cuDF PR for review. At least one of the developers shall be different from the developer who designed and implemented the column abstraction.
• Any required design changes exposed in refactoring shall be discussed in the PR
• Stakeholders will review refactored feature until consensus is reached
• TBD will be responsible for creating/amending a style guide with lessons learned and best practices for refactoring a feature using gdf_column to the new abstraction(s)
• Proposed Deadline: 0.9 Release

6. Full Refactor

• Remaining libcudf features will be refactored one at a time to use the new column abstraction(s)
• The style guide mentioned above will be distributed to all libcudf developers to provide guidance in this refactoring effort
• This will be an ongoing process that likely will not be fully complete for several releases

Is your feature request related to a problem? Please describe. cudf::size_type is currently an int32_t, which limits column size to two billion elements (MAX_INT). Moreover, it limits child column size to the same. This causes problems, for example, for string columns, where there may be fewer than 2B strings, but the character data to represent them could easily exceed 2B characters.

A 32-bit size was originally chosen to ensure compatibility with Apache Arrow, which dictates that Arrow arrays have a 32-bit size, and that larger arrays are made by chunking into individual Arrays.

Describe the solution you'd like

• Change size_type to be an int64_t.

• Handle compatibility with Arrow by creating arrow chunked arrays in the libcudf to_arrow interface (not yet created), and combine arrow chunked arrays in the libcudf from_arrow interface. This can be dealt with when we create these APIs.

Describe alternatives you've considered

Chunked columns. This would be very challenging -- supporting chunked columns in every algorithm would result in complex distributed algorithms and implementations, where libcudf currently aims to be communication agnostic / ignorant. In other words, a higher level library handles distributed algorithms.

Additional context

A potential downside: @felipeblazing called us brave for considering supporting chunked columns. If we implement this feature request, perhaps he will not consider us quite so brave. :(

There have been a number of requests related to adding column names, either to the column's themselves and/or to tables and their views.

libcudf internals don't use column names, so we need requirements to be driven by users that will make use of the names (cuIO/Spark/cuDF).

For those who need column names, please discuss what you would like to see for column names.

CC @kkraus14 @revans2 @jlowe @j-ieong @shwina

Describe the bug nan_as_null parameter affects output of sort_values.

Steps/Code to reproduce bug

In [22]: df = cudf.DataFrame({'a': cudf.Series([np.nan, 1.0, np.nan, 2.0, np.nan, 0.0], nan_as_null=True)})

In [23]: print(df.sort_values(by='a'))
     a
5  0.0
1  1.0
3  2.0
0
2
4
In [19]: df = cudf.DataFrame({'a': cudf.Series([np.nan, 1.0, np.nan, 2.0, np.nan, 0.0], nan_as_null=False)})

In [20]: print(df.sort_values(by='a'))
     a
0  nan
1  1.0
2  nan
3  2.0
4  nan
5  0.0

similar issues with methods using libcudf APIs. Eg. drop_duplicates (which uses sorting)

df = cudf.DataFrame({'a': cudf.Series([1.0, np.nan, 0, np.nan, 0, 1], nan_as_null=False)})

In [10]: print(df)
     a
0  1.0
1  nan
2  0.0
3  nan
4  0.0
5  1.0

In [11]: print(df.drop_duplicates())
     a
0  1.0
1  nan
2  0.0
3  nan
4  0.0
5  1.0

Expected behavior For sorting, drop_duplicates, nan should be considered equal.

Environment overview (please complete the following information)

• Environment location: Bare-metal
• Method of cuDF install: from source

Recent issues (https://github.com/rapidsai/cudf/issues/4753 https://github.com/rapidsai/cudf/issues/4752) have called into question how libcudf handles NaN floating point values. We've only ever addressed this issue on an ad hoc basis as opposed to having a larger conversation about the issue.

C++

C++ follows the IEEE 754 standard for floating point values, which for comparisons with NaN has the following behavior:

| Comparison | NaN ≥ x | NaN ≤ x | NaN > x | NaN < x | NaN = x | NaN ≠ x | |------------|--------------|--------------|--------------|--------------|--------------|-------------| | Result | Always False | Always False | Always False | Always False | Always False | Always True |

https://en.wikipedia.org/wiki/NaN

Spark

Spark is non-conforming with the IEEE 754 standard:

| Comparison | NaN ≥ x | NaN ≤ x | NaN > x | NaN < x | NaN = x | NaN ≠ x | |------------|-------------|-----------------------|-------------|--------------|------------------|----------------------| | Result | Always True | False unless x is NaN | Always True | Always False | True only if x is NaN | True unless x is NaN |

See https://spark.apache.org/docs/latest/sql-reference.html#nan-semantics

Python/Pandas

Python is a bit of a grey area because prior to 1.0, Pandas did not have the concept of "null" values and used NaN's in their stead.

In most regards, Python does respect IEEE 754. For example, see how numpy conforms with the expected IEEE754 behavior in binary ops https://github.com/rapidsai/cudf/issues/4752#issuecomment-606649251 (where Spark does not).

However, there are some cases where Pandas is non-conforming due to the pseudo-null behavior. For example, in sort_values there is a na_position argument to control where NaN values are placed. This requires specializing the libcudf comparator used for sorting to special case floating point values and deviate from the IEEE 754 behavior of NaN < x == false and NaN > x == false. See https://github.com/rapidsai/cudf/issues/2191 and https://github.com/rapidsai/cudf/issues/3226 where this was done previously.

That said, I believe Python's requirements could be satisfied by always converting NaN values to nulls, but @shwina @kkraus14 will need to confirm. Prior to Pandas 1.0, it wasn't possible to have both NaN and NULL values in a floating point column. We should see what the expected behavior is of NaNs vs Nulls will be in 1.0.

Discussion

We need to have a conversation and make decisions on what libcudf will and will not do with respect to NaN behavior.

My stance is that libcudf should adhere to IEEE 754. Spark's semantics redefine a core concept of the C++ language/IEEE standard and satisfying those semantics would require extremely invasive changes that negatively impact both performance and code maintainability.

Even worse, because Spark differs from C++/Pandas, we need to provide separate code paths for all comparison based operations: a "Spark" path, and a "C++/Pandas" path. This further increases code bloat and maintenance costs.

Furthermore, for consistency, I think we should roll back the non-conformant changes introduced for comparators in https://github.com/rapidsai/cudf/issues/3226.

In conclusion, we already have special logic for handling NULLs everywhere in libcudf. Users should leverage that logic by converting NaNs to NULLs. I understand that vanilla Spark treats NaNs and NULLs independently, but I believe trying to imitate that behavior in libcudf comes at too high a cost.

Describe the bug Building v0.9.0 from source fails as some dependencies are missing or not fetched.

Steps/Code to reproduce bug

• git clone and checkout v0.9.0.
• update submodules
• bash build.sh libcudf

-- RMM: RMM_LIBRARY set to RMM_LIBRARY-NOTFOUND
-- RMM: RMM_INCLUDE set to RMM_INCLUDE-NOTFOUND
-- DLPACK: DLPACK_INCLUDE set to DLPACK_INCLUDE-NOTFOUND
-- NVSTRINGS: NVSTRINGS_INCLUDE set to NVSTRINGS_INCLUDE-NOTFOUND
-- NVSTRINGS: NVSTRINGS_LIBRARY set to NVSTRINGS_LIBRARY-NOTFOUND
-- NVSTRINGS: NVCATEGORY_LIBRARY set to NVCATEGORY_LIBRARY-NOTFOUND
-- NVSTRINGS: NVTEXT_LIBRARY set to NVTEXT_LIBRARY-NOTFOUND

Expected behavior Build succeeds without missing deps.

Environment overview (please complete the following information)

• Environment location: Centos 7, avx512
• Method of cuDF install: source

Additional context

• The documentation does not state dlpack, rmm or nvstrings as dependencies.
• According to the RMM README

RMM currently must be built from source. This happens automatically in a submodule when you build or install cuDF or RAPIDS containers.

Users should then expect that theses deps should be automatically pulled.

Hi Developers,

Thanks for the great tools you have made. Our group would like to use cudf for deep learning, however pytorch currently only support CUDA 10.2 and CUDA 11.1, the nightly version rapids supported is CUDA 11.0 and 11.2, which is a pain for users (mostly scientist) if they need to compile either pytorch or rapids from source. Is that possible for rapids to support CUDA 11.1 for user to install from conda?

I noticed that #8224 just remove cuda 11.1 related files.

Thanks! Richard

Adds optional cuFile integration:

• cufile.h is included in the build when available.
• libcufile.so is loaded at runtime if LIBCUDF_CUFILE_POLICY environment variable is set to "ALWAYS" or "GDS".
• cuFile compatibility mode is set through the same policy variable - "ALWAYS" means on, "GDS" means off.
• cuFile is currently only used on Parquet R/W and in CSV writer.
• device_read/write API can be used with file datasource/data_sink.
• Added CUDA stream to device_read.

Description

This PR fixes a final few warnings introduced in PRs merged after #12406 or with changes that weren't merged into #12406. More importantly, this PR updates the testing suite to always raise uncaught warnings as errors. This is a substantial change to the testing suite that will allow us to proactively react to deprecations in our dependencies, identify areas where pandas/pyarrow/some other dependency handles input data in different ways than us, etc. I've opted for the strictest possible setting, but am open to reviewer requests to be a little more lax with certain classes of warnings.

Checklist

• [x] I am familiar with the Contributing Guidelines.
• [x] New or existing tests cover these changes.
• [x] The documentation is up to date with these changes.

Description

Resolves #12303 This PR removes int32 hard-coding which are related to size_type in libcudf. Some of the hard-coding's have been left because they are un-related to size_type.

Checklist

• [x] I am familiar with the Contributing Guidelines.
• [ ] New or existing tests cover these changes.
• [x] The documentation is up to date with these changes.

Description

This uses cudaMemcpyDefault instead of cudaMemcpy{Host,Device}To{Host,Device}. After consultation with @vuule on another PR and asking the CUDA team, there is no additional cost to using the default value over specifying the host/device residency.

The only potential advantage of specifying the direction is that an error is raised if the direction does not match the source/destination parameters' location. @vuule and I are both +1 for this change but neither of us feel strongly if others feel differently. On the other hand, there is less to think about if we always use cudaMemcpyDefault.

Checklist

• [x] I am familiar with the Contributing Guidelines.
• [x] New or existing tests cover these changes.
• [x] The documentation is up to date with these changes.

Describe the bug For an arrow table that contains string columns and is converted from pandas, the from_arrow fails after slice because the length does not match.

Steps/Code to reproduce bug

import cudf
import pyarrow
import pandas as pd

cdf = pd.DataFrame.from_dict({'a': ['aa', 'bb', 'cc'], 'b': [1, 2, 3]})
print(cdf)

tbl = pyarrow.Table.from_pandas(cdf)
print(tbl)

tbl_slice = tbl.slice(0, 2)
print(tbl_slice)

gdf = cudf.DataFrame.from_arrow(tbl_slice)

>>> import cudf
>>> import pyarrow
>>> import pandas as pd
>>>
>>>
>>> cdf = pd.DataFrame.from_dict({'a': ['aa', 'bb', 'cc'], 'b': [1, 2, 3]})
>>> print(cdf)
    a  b
0  aa  1
1  bb  2
2  cc  3
>>>
>>>
>>> tbl = pyarrow.Table.from_pandas(cdf)
>>> print(tbl)
pyarrow.Table
a: string
b: int64
----
a: [["aa","bb","cc"]]
b: [[1,2,3]]
>>>
>>>
>>> tbl_slice = tbl.slice(0, 2)
>>> print(tbl_slice)
pyarrow.Table
a: string
b: int64
----
a: [["aa","bb"]]
b: [[1,2]]
>>>
>>>
>>> gdf = cudf.DataFrame.from_arrow(tbl_slice)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/huawei/release/bi/uxdf/envPkg/Miniconda3/envs/uxdf_server/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/huawei/release/bi/uxdf/envPkg/Miniconda3/envs/uxdf_server/lib/python3.9/site-packages/cudf/core/dataframe.py", line 4458, in from_arrow
    out = out.set_index(
  File "/opt/huawei/release/bi/uxdf/envPkg/Miniconda3/envs/uxdf_server/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/huawei/release/bi/uxdf/envPkg/Miniconda3/envs/uxdf_server/lib/python3.9/site-packages/cudf/core/dataframe.py", line 2453, in set_index
    df.index = idx
  File "/opt/huawei/release/bi/uxdf/envPkg/Miniconda3/envs/uxdf_server/lib/python3.9/site-packages/cudf/core/dataframe.py", line 1027, in __setattr__
    super().__setattr__(key, col)
  File "/opt/huawei/release/bi/uxdf/envPkg/Miniconda3/envs/uxdf_server/lib/python3.9/site-packages/cudf/core/indexed_frame.py", line 341, in index
    raise ValueError(
ValueError: Length mismatch: Expected axis has 2 elements, new values have 3 elements
>>>

Expected behavior Translate correctly.

Environment overview (please complete the following information)

• Environment location: Cloud(HuaweiCloud)
• Method of cuDF install: conda

Environment details Not found.

Hi. Pip installation on Google Colab with !pip install cudf-cu11 --extra-index-url=https://pypi.ngc.nvidia.com seems to work but results in missing functionality. For example, trying to compute column correlations in a dataframe with a min_periods argument specified raises NotImplementedError: Unsupported argument 'min_periods'. Other general functionality seems to be missing as well with various errors raised. Interestingly, everything seemed to be working yesterday. Any thoughts on what could be going on? Having a working pip installation on colab would be a game changer!

Description

Fixes #12456.

This is purely a JNI change. The implementation of jni_writer_data_sink has been streamlined a little:

1. The commonality in device_write() and host_write() has been moved to a common function.
2. rotate_buffer() has been renamed to handle_buffer_and_reallocate().
3. jni_writer_data_sink has been moved to the cudf::jni::io namespace.
4. jni_writer_data_sink is now named writer_data_sink.

Checklist

• [x] I am familiar with the Contributing Guidelines.
• [x] New or existing tests cover these changes.
• [x] The documentation is up to date with these changes.

Note: This is purely a JNI change. This is a follow-up to #12425, which has yet to be merged. The changes in that PR are included in this one. Once #12425 is merged, this changes here will be revealed as minuscule.