Data Import

Catalog data enters through a package of CSV files and a run of manage.py commands in a fixed order. The commands and their order below are the ones the Emporium test package runs on every seed.

What Gets Imported

A package is a directory of CSV files, with Django YAML fixtures next to it. PIM is the source of truth for products, categories and attributes. Prices live in the price manager. Stock arrives through QMS, which reads a quantity file and pushes it into checkout stock. Matrix then builds the read model from all three, and that read model is what the storefront API serves. Terms such as channel and idx are defined in Concepts.

The Package

File names follow <kind>--<key>.csv, where the key is a channel idx, a feature or a delivery point type.

File	Read by	Keyed by
volkanos-config/channels.conf	the import script, to drive every per-channel step	channel:currency lines
volkanos-config/pim-channels.csv	config-load-pim-channels	idx
volkanos-config/pim-features.csv	config-load-pim-features	feature idx
volkanos-config/pim-features-sets.csv	config-load-pim-features-sets	feature set and feature
volkanos-config/pim-feature-position-in-features-sets.csv	config-load-pim-feature-position-in-features-sets	feature set and feature
attributes--<feature>.csv	attributes-import-from-csv	idx
categories--<channel>.csv	categories-import-from-csv	idx
products--<channel>.csv	products-import-from-csv	sku
products-position--<channel>.csv	products-position-import-from-csv	sku and category_idx
pricelist--<channel>.csv	import-pricelist-from-csv	sku
qty.csv	QMS, through qms-manage-quantities	sku
deliverypoints--<type>.csv	import_deliverypoints	delivery-point-code

Header rows, quoted from the Emporium package:

volkanos-config/pim-channels.csv
idx,name,default_language,languages,default_currency,currencies

products--default-europe.csv
sku,feature set,name pl,name en,category,options,series,brand,badge,url_key pl,url_key en,dimensions_depth,dimensions_height,dimensions_width,customizable,product type,config features,config sku,bundle sku,bundle idx,feature set custom,max_limit_bundle,min_limit_bundle,weight_kg,is_handcrafted,material_composition,specifications,short_description en,description en,img main,img 1,img 2,images

pricelist--default-europe.csv
sku,tax_class,gross,special_price_gross

qty.csv
sku,quantity

The full column reference for the PIM files is csv-format.md.

Two directories sit next to package/: fixtures/ with the Django YAML fixtures, and images/ with product pictures. When images/ is missing, the product import runs with --skip-pictures and thumbnail generation is skipped.

Channels and Languages

Every module that cares about channels has its own channel records, and they are loaded from the fixtures before any CSV is read: django_regional (currencies and languages), django_pim channels, django_pricemanager channels and sale channels, django_qms, django_checkout and django_matrix channels. The seed loads them as its fixtures step, from a fixed list of fixture files.

Then config-load-pim-channels reads pim-channels.csv, with the columns idx,name,default_language,languages,default_currency,currencies.

channels.conf decides which channels the per-channel steps run for. Each line is channel:currency:

default-local:USD
default-europe:EUR

The IMPORT_CHANNELS environment variable overrides the file, as comma separated channel:currency pairs.

A few modules copy their channels and languages from PIM after the import. The seed runs these after the package import:

python manage.py sync_contentdb_languages
python manage.py sync_contentdb_channels
python manage.py sync_faq_channels
python manage.py sync_dp_channels

Step by Step

Each command belongs to one module:

Command	Module
config-load-pim-*, attributes-import-from-csv, categories-import-from-csv, products-import-from-csv, products-position-import-from-csv	pim-csv
import-pricelist-from-csv, manage-pricelists	pricemanager
pim-thumbs-generate	pim
fill-omnibus-product-representation-from-pim, calculate-omnibus-price	omnibus
qms-manage-quantities	qms
fill-read-model	matrix
import_deliverypoints	deliverypoints

The Check lines use the storefront API v2 on the default-europe channel. Run the steps in this order: later steps look up what earlier ones created.

PIM configuration

• Command:

  python manage.py config-load-pim-channels package/volkanos-config/pim-channels.csv
  python manage.py config-load-pim-features package/volkanos-config/pim-features.csv
  python manage.py config-load-pim-features-sets package/volkanos-config/pim-features-sets.csv --prune
  python manage.py config-load-pim-feature-position-in-features-sets package/volkanos-config/pim-feature-position-in-features-sets.csv --prune

• Reads: the four files in volkanos-config/.
• Creates: PIM channels, features, feature sets and feature positions.
• Check: any rejected row fails the command and imports nothing. --prune detaches what the CSV does not list. The seed then reruns both --prune commands with --dry-run and expects Would detach 0, which proves the database matches the CSV.

Attributes

• Command: python manage.py attributes-import-from-csv brand package/attributes--brand.csv, once per feature (the package has badge, brand, series and options).
• Reads: attributes--<feature>.csv.
• Creates: PIM attribute values for that feature.
• Check: the products listing call below returns the attribute values on each product once the read model is filled.

Categories

• Command: python manage.py categories-import-from-csv default-europe package/categories--default-europe.csv
• Reads: categories--<channel>.csv, one per channel.
• Creates: the PIM category tree of that channel.
• Check: curl "http://localhost:8100/api/matrix/v2/default-europe/categories/?language=en" once the read model is filled.

Products

• Command: python manage.py products-import-from-csv default-europe package/products--default-europe.csv --skip-pictures
• Reads: products--<channel>.csv, and images/ when it exists.
• Creates: PIM products with their translations, attributes and pictures. --skip-pictures is passed only when there is no images/ directory.
• Check: curl "http://localhost:8100/api/matrix/v2/default-europe/products/?language=en&currency=EUR" once the read model is filled.

Product positions

• Command: python manage.py products-position-import-from-csv default-europe package/products-position--default-europe.csv
• Reads: products-position--<channel>.csv.
• Creates: the order of products inside each PIM category.
• Check: the products listing call above, once the read model is filled.

Prices

• Command:

  python manage.py import-pricelist-from-csv default-europe package/pricelist--default-europe.csv --currency_code=EUR
  python manage.py manage-pricelists

• Reads: pricelist--<channel>.csv; the currency comes from channels.conf.
• Creates: price manager price lists and prices.
• Check: the products listing call with currency=EUR returns prices once the read model is filled.

The seed then gives bundle components their own price inside each bundle with a short manage.py shell snippet; it is part of the test package, not a product command.

Thumbnails

• Command: python manage.py pim-thumbs-generate default-europe
• Reads: the pictures imported with the products.
• Creates: PIM picture thumbnails. Runs only when pictures were imported.
• Check: the products listing call returns image URLs once the read model is filled.

Omnibus

• Command:

  python manage.py fill-omnibus-product-representation-from-pim default-europe
  python manage.py calculate-omnibus-price

• Reads: PIM products and price manager prices.
• Creates: omnibus product representations and the lowest prior price per product.
• Check: the products listing call, once the read model is filled.

Quantities

• Command: python manage.py qms-manage-quantities
• Reads: qty.csv, through the xraystorage records of the QMS fixture, which point at the file.
• Creates: QMS quantities, pushed into django_checkout stock.
• Check: the products listing call returns stock once the read model is filled.

The pull and push run as Celery tasks, so a worker must be up. The command returns before the chain is done; the import script polls until checkout stock settles before it moves on.

Read model

• Command: python manage.py fill-read-model
• Reads: PIM, price manager prices, and stock from django_checkout, falling back to QMS when checkout is not installed.
• Creates: the Matrix read model.
• Check: both curl lines above answer with status 200.

The import script runs qms-manage-quantities and fill-read-model a second time. Configurable products get their checkout records only once the catalog is complete, so the first push can miss them; the second one fills them in.

Delivery points

• Command: python manage.py import_deliverypoints --file package/deliverypoints--dpd.csv --type dpd --mode incremental
• Reads: deliverypoints--<type>.csv; the type is the part of the file name after deliverypoints--.
• Creates: delivery points of that type.
• Check: the delivery points panel of the CMS.

Running the Whole Pipeline

In the zeno harness, clone the test package before the stack starts; it is bind mounted into the containers:

make clone-tests
make seed

make seed drops and recreates the database, loads the fixtures, creates the admin user with password admin123, runs the import, runs the syncs, and ends with SEED OK. A failure prints SEED FAILED (exit N) during: <step>.

To run the import alone, open a shell in the service container with make shell. The script needs SVC_DIR, the directory of the service; the container has it as SERVICE_DIR:

SVC_DIR="$SERVICE_DIR" bash /entirius/test-package/scripts/import-package.sh /entirius/test-package/package

Your Own Package

A package can be generated from a profile and a catalog file, as Your Data describes, or built by hand:

1. Copy package/ and fixtures/ from the Emporium test package.
2. Keep the file names: <kind>--<channel>.csv.
3. Edit channels.conf and pim-channels.csv first; every other file depends on the channels.
4. Keep idx values stable. They are the keys the importers update by, so a re-import updates records in place.
5. For incremental changes use --update-only or, for products, --update-translations-only. The modes are described in import-pipeline.md.

The seed script belongs to the test package, not to the product. A service without the harness runs the manage.py commands of Step by Step in the same order.

Next

Add what no existing module covers: Build a Module.