# MaxAbsScaler to process data, and the processing speed is extremely slow

**URL:** <https://discuss.ray.io/t/maxabsscaler-to-process-data-and-the-processing-speed-is-extremely-slow/12016>\
**Category:** Uncategorized\
**Created:** [September 4, 2023, 2:36am UTC](https://discuss.ray.io/t/maxabsscaler-to-process-data-and-the-processing-speed-is-extremely-slow/12016 "2023-09-04T02:36:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![yiwei00000](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yiwei00000/32/4985_2.png) [@yiwei00000](https://discuss.ray.io/u/yiwei00000)\
**Post date:** [September 4, 2023, 2:36am UTC](https://discuss.ray.io/t/maxabsscaler-to-process-data-and-the-processing-speed-is-extremely-slow/12016/1 "2023-09-04T02:36:06Z")

</div>

### What happened + What you expected to happen

Our data volume is 100000 pieces, each with 1000 columns, totaling approximately 1.5G. The data is stored on hdfs.  
The Ray cluster has 6 cores, each with 24GB of memory, and a total of three nodes.

I use MaxAbsScaler’s fit\_transform It takes about 10 minutes for the transformer to process these data, but I only need 15 seconds to use Spark’s MaxAbsScaler to fit and transform. Why is ray so slow? Spark and ray use the same machine resources.

### Versions / Dependencies

ray 2.6.0  
hdfs 3.2.2

### Reproduction script

import ray  
from pyarrow import fs

ray.init(‘auto’)  
hdfs\_fs = fs.HadoopFileSystem.from\_uri(“hdfs://jd-hadoop/?user=root”)  
ds = ray.data.read\_csv(‘/data/eps-files/’, filesystem=hdfs\_fs,parallelism=2000)  
files = sorted(ds.input\_files())  
ds = ray.data.read\_csv(files[0:100], filesystem=hdfs\_fs,parallelism=100)  
ds = ds.materialize()  
ds.max(“x1”)

cols = ds.columns()[0:1001]  
ds\_1001 = ds.select\_columns(cols)  
ds\_1001 = ds\_1001.materialize()  
from ray.data.preprocessors import MaxAbsScaler  
preprocessor = MaxAbsScaler(columns=cols[1:len(cols)])  
ds\_1001\_trans = preprocessor.fit\_transform(ds\_1001) # **It takes about 10 minutes**  
ds\_1001\_trans = ds\_1001\_trans.materialize()  
ds\_1001\_trans.show(10)

### Issue Severity

High: It blocks me from completing my task.