Your Data

A catalog of your own becomes importable in two files: a profile of the channel and a catalog file with the categories and the products. One command turns them into a data package.

What a Data Package Is

A data package is a directory with package/ (the CSV files), fixtures/ (the channel records of each module), images/ (pictures by SKU), first-channel.toml and manifest.json. All of it is generated from the two files you write. The file formats and the order of the import are on Data Import.

The generator is part of the Emporium test package. It reads only your files and downloads nothing.

The Profile

profile.yaml describes one channel:

# The profile of one channel. Every key under `channel` is required.
channel:
  idx: demo-shop            # the key of the channel in every module and in the CSV file names
  name: Demo Shop           # the name of the channel as the CMS shows it
  currency: EUR             # ISO 4217, upper case
  default_language: en      # ISO 639-1, lower case; one of `languages`
  languages: [en, de]       # every language gets its own name, URL key and description columns
  default_country: DE       # ISO 3166-1 alpha-2, upper case
  tax_rate: "0.19"          # the standard rate of default_country, as a decimal string
sku_prefix: DEMO            # products without a sku get <prefix>-P001, -P002 … in catalog order
default_quantity: 20        # stock for products without a quantity

idx is the key of the channel everywhere: in every module, in the file names and in the API paths. See Concepts.
currency, the languages and default_country must be codes the regional module ships.
tax_rate is yours to set. The generator does not look it up.
sku_prefix and default_quantity are optional. They apply to products that carry no SKU or no quantity.

The Catalog File

The catalog is one file, JSON or CSV. The generator picks the format by the file suffix.

In JSON, categories are a tree and each product names its category by slug:

{
  "categories": [
    {
      "name": "Tea",
      "slug": "tea",
      "children": [
        {
          "name": "Green Tea",
          "slug": "green-tea"
        }
      ]
    }
  ],
  "products": [
    {
      "sku": "DEMO-TEA-1",
      "name": "Black Tea 100 g",
      "slug": "black-tea-100g",
      "price": "6.50",
      "quantity": 40,
      "brand": "Brand A",
      "category_slug": "tea",
      "description": "Loose leaf black tea.",
      "images": [
        "images/black-tea.png"
      ]
    }
  ]
}

Product key	Required	Holds
`name`	yes	the product name
`price`	yes	the price, as a decimal string
`category_slug`	yes	the slug of one category of the file
`sku`	no	the SKU; without it the product gets one from `sku_prefix`
`slug`	no	the URL key; without it the name is turned into one
`quantity`	no	the stock; without it `default_quantity` applies
`brand`	no	the brand name
`description`	no	the product description
`images`	no	a list of image files, the main picture first

In CSV, one row is one product:

sku,name,slug,price,quantity,brand,category,category_name,parent_category,description,images

The categories come from three columns: category is the slug, category_name its name and parent_category the slug of the parent, empty for a top-level category. Several images of one product are separated with |.

Images are local files, with paths relative to the catalog file. A URL is refused.

Generate the Package

In a checkout of the test package, one command builds the package and validates it:

make data-package PROFILE=examples/data-package/profile.yaml CATALOG=examples/data-package/catalog.json OUT=/tmp/demo-shop

It runs two steps, which also work on their own:

uv run --extra data python -m entirius_tests.data_package build --profile examples/data-package/profile.yaml --catalog examples/data-package/catalog.json --out /tmp/demo-shop

uv run --extra data python -m entirius_tests.data_package validate /tmp/demo-shop

build prints one line per file it writes:

data-package: wrote first-channel.toml
data-package: wrote fixtures/django_accounts.cfg.yaml
…
data-package: wrote package/volkanos-config/pim-features.csv
data-package: wrote manifest.json
data-package: done demo-shop -> /tmp/demo-shop

validate reads the package back and prints one line:

data-package: valid demo-shop (4 categories, 6 products, 4 images)

The output directory must be new or empty. For the example it looks like this:

.
├── first-channel.toml
├── fixtures
│   ├── django_accounts.cfg.yaml
│   ├── django_checkout.cfg.yaml
│   ├── django_matrix.cfg.yaml
│   ├── django_pim.cfg.yaml
│   ├── django_pricemanager.cfg.yaml
│   ├── django_qms.cfg.yaml
│   └── django_regional.cfg.yaml
├── images
│   ├── DEMO-COF-1
│   │   └── 1.png
│   ├── DEMO-TEA-1
│   │   └── 1.png
│   └── DEMO-TEA-2
│       ├── 1.png
│       └── 2.png
├── manifest.json
└── package
    ├── attributes--brand.csv
    ├── categories--demo-shop.csv
    ├── pricelist--demo-shop.csv
    ├── products--demo-shop.csv
    ├── qty.csv
    └── volkanos-config
        ├── channels.conf
        ├── pim-channels.csv
        ├── pim-feature-position-in-features-sets.csv
        ├── pim-features.csv
        └── pim-features-sets.csv

build also takes --mount: the path the package has where the service and the worker read it. It defaults to the absolute path of the output directory. The QMS fixture stores the path of qty.csv, so a package that is copied elsewhere for the import is built with the path it will have there.

Load the Package

The package fits an empty, migrated database. The commands run in the service, with the package at the path it was built for.

The fixtures come first, in the order manifest.json lists them:

python manage.py loaddata --format=yaml /tmp/demo-shop/fixtures/django_regional.cfg.yaml
python manage.py loaddata --format=yaml /tmp/demo-shop/fixtures/django_pim.cfg.yaml
python manage.py loaddata --format=yaml /tmp/demo-shop/fixtures/django_pricemanager.cfg.yaml
python manage.py loaddata --format=yaml /tmp/demo-shop/fixtures/django_qms.cfg.yaml
python manage.py loaddata --format=yaml /tmp/demo-shop/fixtures/django_checkout.cfg.yaml
python manage.py loaddata --format=yaml /tmp/demo-shop/fixtures/django_matrix.cfg.yaml
python manage.py loaddata --format=yaml /tmp/demo-shop/fixtures/django_accounts.cfg.yaml

Then the PIM configuration, the attributes, the categories and the products, with the commands of Step by Step. Three things differ for a generated package: the files sit under /tmp/demo-shop/package/, the channel idx of the profile takes the place of default-europe, and brand is the only attributes file. Run the product import with --skip-pictures, as Step by Step shows it.

Last, the first-channel script of the service, with the generated config:

python scripts/first_channel.py /tmp/demo-shop/first-channel.toml

The fixtures already created the channel in every module, so the script reports exists for those and adds the rest: the sync of the other modules, the storefront key and the module registry. The output, line by line: The First Channel.

With --skip-pictures the product import leaves the package’s images/ out. The import script of Data Import imports them: it drops the flag when images/ sits next to package/ and generates the thumbnails afterwards. Prices, quantities and the read model follow in the order of Step by Step.

Check the Result

Before the import, validate checks the package: every listed file exists, the channel is the same everywhere, and every product has its category, a price row, a quantity row and its image files.

After the import, the configuration and import commands each end with Command finished with success, and the first-channel script ends with done and the idx of the channel.

The products are in the product list of the channel in the CMS once the product import ran: Products. The storefront shows them once the read model is filled, and the check calls are on Data Import. Both must point at the channel, as Your Own Service describes.

The file formats, the full import order and re-imports: Data Import.

Previous
Your Own Service Next
Data Import