.. _sec_imgdataset:

Image Prediction - Properly load any image dataset as ImageDataset
==================================================================


Preparing the dataset for ImagePredictor is not difficult at all,
however, we’d like to introduce the recommended ways to initialize the
dataset, so you will have a smoother experience using
``autogluon.vision.ImagePredictor``.

There are generally three ways to load a dataset for ImagePredictor:

-  Load a csv file or construct your own pandas ``DataFrame`` with
   ``image`` and ``label`` columns

-  Load a image folder directly with ``ImageDataset``

-  Convert a list of images into a dataset directly with
   ``ImageDataset``

We will go through these four methods one by one. First of all, let’s
import AutoGluon:

.. code:: python

    %matplotlib inline
    import autogluon.core as ag
    from autogluon.vision import ImageDataset
    import pandas as pd


.. parsed-literal::
    :class: output

    /home/ci/opt/venv/lib/python3.8/site-packages/gluoncv/__init__.py:40: UserWarning: Both `mxnet==1.9.1` and `torch==1.12.1+cu102` are installed. You might encounter increased GPU memory footprint if both framework are used at the same time.
      warnings.warn(f'Both `mxnet=={mx.__version__}` and `torch=={torch.__version__}` are installed. '
    INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm71k5ag0
    INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm71k5ag0/_remote_module_non_scriptable.py
    INFO:root:Generating grammar tables from /usr/lib/python3.8/lib2to3/Grammar.txt
    INFO:root:Generating grammar tables from /usr/lib/python3.8/lib2to3/PatternGrammar.txt


Load a csv file or construct a DataFrame object
-----------------------------------------------

We use a csv file from PetFinder competition as an example. You may use
any tabular data as long as you can create ``image``\ (absolute or
relative paths to images) and ``label``\ (category for each image)
columns.

.. code:: python

    csv_file = ag.utils.download('https://autogluon.s3-us-west-2.amazonaws.com/datasets/petfinder_example.csv')
    df = pd.read_csv(csv_file)
    df.head()


.. parsed-literal::
    :class: output

    Downloading petfinder_example.csv from https://autogluon.s3-us-west-2.amazonaws.com/datasets/petfinder_example.csv...
    100%|██████████| 820/820 [00:00<00:00, 2638.92KB/s]




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>image</th>
          <th>PetID</th>
          <th>label</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>petfinder_data/train_images/015da9e87-1.jpg</td>
          <td>015da9e87</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>petfinder_data/train_images/022606901-1.jpg</td>
          <td>022606901</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>petfinder_data/train_images/02f89bdcb-1.jpg</td>
          <td>02f89bdcb</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>petfinder_data/train_images/03f217352-1.jpg</td>
          <td>03f217352</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>petfinder_data/train_images/040a9a6f9-1.jpg</td>
          <td>040a9a6f9</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



If the image paths are not relative to the current working directory,
you may use the helper function to prepend a prefix for each image.
Using absolute paths can reduce the chance of an OSError happening
during file access:

.. code:: python

    df = ImageDataset.from_csv(csv_file, root='/home/ubuntu')
    df.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>image</th>
          <th>PetID</th>
          <th>label</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>/home/ubuntu/petfinder_data/train_images/015da...</td>
          <td>015da9e87</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>/home/ubuntu/petfinder_data/train_images/02260...</td>
          <td>022606901</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>/home/ubuntu/petfinder_data/train_images/02f89...</td>
          <td>02f89bdcb</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>/home/ubuntu/petfinder_data/train_images/03f21...</td>
          <td>03f217352</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>/home/ubuntu/petfinder_data/train_images/040a9...</td>
          <td>040a9a6f9</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



Or you can perform the correction by yourself:

.. code:: python

    import os
    df['image'] = df['image'].apply(lambda x: os.path.join('/home/ubuntu', x))
    df.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>image</th>
          <th>PetID</th>
          <th>label</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>/home/ubuntu/petfinder_data/train_images/015da...</td>
          <td>015da9e87</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>/home/ubuntu/petfinder_data/train_images/02260...</td>
          <td>022606901</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>/home/ubuntu/petfinder_data/train_images/02f89...</td>
          <td>02f89bdcb</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>/home/ubuntu/petfinder_data/train_images/03f21...</td>
          <td>03f217352</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>/home/ubuntu/petfinder_data/train_images/040a9...</td>
          <td>040a9a6f9</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



Otherwise you may use the ``DataFrame`` as-is, ``ImagePredictor`` will
apply auto conversion during ``fit`` to ensure other metadata is
available for training. You can have multiple columns in the
``DataFrame``, ``ImagePredictor`` only cares about ``image`` and
``label`` columns during training.

Load an image directory
-----------------------

It’s pretty common that sometimes you only have a folder of images,
organized by the category names. Recursively looping through images is
tedious. You can use ``ImageDataset.from_folders`` or
``ImageDataset.from_folder`` to avoid implementing recursive search.

The difference between ``from_folders`` and ``from_folder`` is the
targeting folder structure. If you have a folder with splits, e.g.,
``train``, ``test``, like:

-  root/train/car/0001.jpg
-  root/train/car/xxxa.jpg
-  root/val/bus/123.png
-  root/test/bus/023.jpg

Then you can load the splits with ``from_folders``:

.. code:: python

    train_data, _, test_data = ImageDataset.from_folders('https://autogluon.s3.amazonaws.com/datasets/shopee-iet.zip', train='train', test='test')
    print('train #', len(train_data), 'test #', len(test_data))
    train_data.head()


.. parsed-literal::
    :class: output

    data/
    ├── test/
    └── train/
    train # 800 test # 80




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>image</th>
          <th>label</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



If you have a folder without ``train`` or ``test`` root folders, like:

-  root/car/0001.jpg
-  root/car/xxxa.jpg
-  root/bus/123.png
-  root/bus/023.jpg

Then you can load the splits with ``from_folder``:

.. code:: python

    # use the train from shopee-iet as new root
    root = os.path.join(os.path.dirname(train_data.iloc[0]['image']), '..')
    all_data = ImageDataset.from_folder(root)
    all_data.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>image</th>
          <th>label</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>2</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>/home/ci/.gluoncv/datasets/shopee-iet/data/tra...</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    </div>



.. code:: python

    # you can manually split the dataset or use `random_split`
    train, val, test = all_data.random_split(val_size=0.1, test_size=0.1)
    print('train #:', len(train), 'test #:', len(test))


.. parsed-literal::
    :class: output

    train #: 653 test #: 82


Convert a list of images to dataset
-----------------------------------

You can create a dataset from a list of images with a function, the
function is used to determine the label of each image. We use the
Oxford-IIIT Pet Dataset mini pack as an example, where images are
scattered in ``images`` directory but with a unique pattern: filenames
of cats start with a capital letter, otherwise they are dogs. So we can
use a function to distinguish and assign a label to each image:

.. code:: python

    pets = ag.utils.download('https://autogluon.s3-us-west-2.amazonaws.com/datasets/oxford-iiit-pet-mini.zip')
    pets = ag.utils.unzip(pets)
    image_list = [x for x in os.listdir(os.path.join(pets, 'images')) if x.endswith('jpg')]
    def label_fn(x):
        return 'cat' if os.path.basename(x)[0].isupper() else 'dog'
    new_data = ImageDataset.from_name_func(image_list, label_fn, root=os.path.join(os.getcwd(), pets, 'images'))
    new_data


.. parsed-literal::
    :class: output

    Downloading oxford-iiit-pet-mini.zip from https://autogluon.s3-us-west-2.amazonaws.com/datasets/oxford-iiit-pet-mini.zip...
    100%|██████████| 35730/35730 [00:01<00:00, 21014.09KB/s]




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>image</th>
          <th>label</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>1</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>2</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>3</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>4</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>...</th>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th>95</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>96</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>97</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>98</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>99</th>
          <td>/home/ci/autogluon/docs/_build/eval/tutorials/...</td>
          <td>0</td>
        </tr>
      </tbody>
    </table>
    <p>100 rows × 2 columns</p>
    </div>



Visualize images
----------------

You can use ``show_images`` to visualize the images, as well as the
corresponding labels:

.. code:: python

    new_data.show_images()



.. figure:: output_dataset_4aa41c_16_0.png


For raw DataFrame objects, you can convert them to Dataset first to use
``show_images``.

Congratulations, you can now proceed to :ref:`sec_imgquick` to start
training the ``ImagePredictor``.