# How to use Ray Tune for Stratified K-Fold Cross-validation on my binary image classification model?
**URL:** https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783
**Category:** Ray Tune
**Created:** [May 24, 2023, 7:57pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783 "2023-05-24T19:57:35Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@chokevin8](https://discuss.ray.io/u/chokevin8)
#### Post date: [May 24, 2023, 7:57pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/1 "2023-05-24T19:57:36Z")
Hi, this is my first time trying to use Ray Tune to tune my hyperparameters for my binary image classification model. I’ve completed training on a stratified 5-fold cross validation scheme, meaning that I have a total of five models for each fold. For each fold, I train for about 10 epochs, and based on the validation metric (F1 score), the best model for the fold is selected and that’s how I end up with 5 models.
Now, how do I perform hyperparameter tuning with the same stratified 5-fold cross validation scheme? I’ve looked everywhere for a pytorch example, but there isn’t any regarding k-fold cross validation. I’d appreciate any help, thank you so much!
---
### Author:  [@gjoliver](https://discuss.ray.io/u/gjoliver)
#### Post date: [May 24, 2023, 11:11pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/2 "2023-05-24T23:11:05Z")
Hi thanks for the question.
How are you building the datasets for each fold today?
are they static datasets or created dynamically at the start of each run?
Are the non-hp-tuning runs done using Ray Tune?
Just trying to understand the problem a bit more. Thanks.
---
### Author:  [@chokevin8](https://discuss.ray.io/u/chokevin8)
#### Post date: [May 25, 2023, 7:06pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/3 "2023-05-25T19:06:37Z")
Thanks for the reply. The dataset is a stratified 5-fold CV training, but they are fixed by setting the random seeds. Therefore, they can be viewed as a static dataset even though they’re “random”. I’m not really sure what you mean about “non-hp-tuning runs”, I haven’t used ray tune in any way yet. Any help would be appreciated, thanks!
---
### Author:  [@gjoliver](https://discuss.ray.io/u/gjoliver)
#### Post date: [May 25, 2023, 7:53pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/4 "2023-05-25T19:53:15Z")
Cool. This is exactly what I was curious about.
Which framework did you use for your 5-fold CV run then? Is it SciKitLearn?
I think before we start talking about using Tune to HP tune your job, we need to replicate your single run job using Ray Library somehow.
It could be through Ray AIR Trainers,
[https://docs.ray.io/en/latest/train/api/api.html](https://docs.ray.io/en/latest/train/api/api.html)
Or it could be a single trainable function that AIR can launch.
Once we get the single run work on Ray, we can easily adapt and make it a mulit-run tuning job.
---
### Author:  [@chokevin8](https://discuss.ray.io/u/chokevin8)
#### Post date: [May 25, 2023, 9:25pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/5 "2023-05-25T21:25:27Z")
Yes, I use the code below, which is sklearn’s StratifiedKFold. By testing a single run job, you’re meaning I should first train the same exact training code I have but just with Ray AIR Trainers? I can go ahead and do that, are there any good examples out there I can follow? My code is based on pytorch. Thanks for your quick replies and help!
> new\_df\_train = train\_df.copy(deep=True)
> strat\_kfold = StratifiedKFold(shuffle = True, random\_state = 42) #use default n\_split = 5, random\_state for reproducibility
>
> #split on white and non-white and add a new column fold to it:
> for each\_fold, (idx1,idx2) in enumerate (strat\_kfold.split(X = new\_df\_train, y = new\_df\_train[‘label’])):
> new\_df\_train.loc[idx2,‘fold’] = int(each\_fold) #create new fold column with the fold number (up to 5)
---
### Author:  [@chokevin8](https://discuss.ray.io/u/chokevin8)
#### Post date: [May 30, 2023, 10:16pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/6 "2023-05-30T22:16:04Z")
@gjoliver Hi, I’d appreciate it if you could get to this whenever you can 🙂 thank you so much!
---
### Author:  [@gjoliver](https://discuss.ray.io/u/gjoliver)
#### Post date: [June 1, 2023, 6:32am UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/7 "2023-06-01T06:32:53Z")
Hi,
here are some hints.
You should be able to write your sklearn job using Ray’s SKLearnTrainer described here:
[https://docs.ray.io/en/latest/train/api/doc/ray.train.sklearn.SklearnTrainer.html](https://docs.ray.io/en/latest/train/api/doc/ray.train.sklearn.SklearnTrainer.html)
to create a stratified train / test split using Ray Dataset, you can take a look at the following example:
```auto
def split_stratify(
ds: Dataset,
stratify: str,
test_size: float = .2
) -> Tuple[Dataset, Dataset]:
'''
Splits a dataset into training and testing subsets according to a
specified test size and optional stratification column.
Args:
ds: The dataset to split.
stratify_by: The name of a column in the dataset to use for stratification.
test_size: The proportion of the dataset to use for testing, between 0 and 1. Defaults to 0.2.
Returns:
Tuple[Dataset, Dataset]: A tuple of two datasets: the training subset and the testing subset.
'''
def split(df: pd.DataFrame) -> pd.DataFrame:
"Data grouped per strata."
# Mark everything 'train' first.
df['__c'] = "train"
# Mark the test split as 'test'.
df.loc[df.tail(int(len(df) * test_size)).index,'__c'] = "test"
return df
splitted = ds.groupby(stratify).map_groups(split)
def filter(split: str):
return splitted.filter(
lambda x: x['__c'] == split
).drop_columns(['__c'])
return filter("train"), filter("test")
train, test = split_stratify(ds, 'F1')
print(train.to_pandas().groupby('F1').count())
```
Let’s start from here and see if you can get a single stratified training job up using Ray Train.
Once we do that, it would be relatively straight forward to introduce hp searching on top of this.
---
### Author:  [@chokevin8](https://discuss.ray.io/u/chokevin8)
#### Post date: [June 1, 2023, 11:06pm UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/8 "2023-06-01T23:06:04Z")
@gjoliver Please excuse me if I’m wrong or misunderstanding, but the ray train’s sklearntrainer means that it runs the fit method of sklearn estimators and it seems like you can do the stratification in the “cv” parameter. However, I’m not sure what you mean when I’m not using any of the sklearn’s estimators to train my DL model.
For the stratification, if I already have the dataset stratified in to five different folds for cross validation (I have this saved) during training (I have stratified folds from 0 to 4, and use fold=fold for validation and fold!= fold for training, so a natural 80-20 split), can I ideally just write run the ray tuner five different times for each model training?
---
### Author:  [@chokevin8](https://discuss.ray.io/u/chokevin8)
#### Post date: [June 5, 2023, 12:34am UTC](https://discuss.ray.io/t/how-to-use-ray-tune-for-stratified-k-fold-cross-validation-on-my-binary-image-classification-model/10783/9 "2023-06-05T00:34:58Z")
@gjoliver Hi I’m so sorry if I’m bothering you, but I would appreciate some guidance, thank you so much!