Aliexpress-Part-DB-orders-pull/README.md

# AliExpress → Part-DB Importer

A two-script toolkit for taking an AliExpress `orders.csv` export and turning
it into a Part-DB v2.15.x inventory with **product images automatically
attached to each part**.

```
┌─────────────────┐    ┌──────────────────────────┐    ┌─────────────┐
│ orders.csv      │ ─► │ aliexpress_to_partdb.py  │ ─► │ partdb_     │
│ (from AliExpr.) │    │  • renames columns       │    │ import.csv  │
└─────────────────┘    │  • downloads all images  │    └──────┬──────┘
                       │  • writes attachments    │           │
                       │    index                 │      [import via
                       └──────────────────────────┘       Part-DB web UI]
                                                            │
                                                            ▼
                       ┌──────────────────────────┐    ┌─────────────┐
                       │ attach_images_to_        │ ◄──│ parts now   │
                       │   partdb.py              │    │ exist in    │
                       │  • reads index           │    │ Part-DB     │
                       │  • finds each part by    │    └─────────────┘
                       │    name via REST API     │
                       │  • uploads image as      │
                       │    attachment            │
                       └──────────────────────────┘
```

---

## What's in this package

| File | Purpose |
|---|---|
| `aliexpress_to_partdb.py` | Reads AliExpress `orders.csv`, downloads item images, writes Part-DB-format CSV |
| `attach_images_to_partdb.py` | Pushes the downloaded images to Part-DB and attaches them via the REST API |
| `README.md` | This file |

Put all three files in the same folder, e.g. `C:\Users\Amir\Downloads\aliexpress-import\`.

---

## Prerequisites

1. **Python 3.8 or newer** — check with `python --version`
2. **The `requests` library** — install with:
   ```cmd
   pip install requests
   ```
3. **Part-DB v2.15.x** running and accessible (test by opening it in your browser)
4. **A Part-DB API token** (see below)
5. **At least one Attachment type** defined in Part-DB (e.g. "Image")
   — see [Step 1.5](#step-15-one-time-create-an-image-attachment-type)

---

## One-time setup

### Step 1: Get an API token in Part-DB

1. Open Part-DB in your browser, log in
2. Click your **username** (top right) → **Settings**
3. In the left sidebar, click **API tokens**
4. Click **+ Create new token** (or "New")
5. Give it any name (e.g. `importer`) and click **Save**
6. **Copy the token** — it looks like `tcp_xxxxxxxxxxxxxxxxxxxxxxxxxx`
   - You won't be able to see it again after this dialog closes
   - If you lose it, just create a new one

> If the "API tokens" section is missing, your user doesn't have the
> **API** permission. If you installed Part-DB yourself you're an admin
> and already have it; otherwise ask your admin to enable
> "Miscellaneous → API access" + "Create API tokens" for your user.

### Step 1.5 (one-time): Create an "Image" attachment type

Part-DB requires an attachment type to exist before any attachment can be created.

1. In Part-DB, click **Admin** in the left sidebar
2. Click **Attachment types**
3. Click **+ New**
4. Name: `Image` (or whatever you prefer)
5. Filetypes: leave blank, or add `image/jpeg`, `image/png` for stricter validation
6. Click **Save**

Verify it worked:
```cmd
python attach_images_to_partdb.py --list-attachment-types
```
You should see your "Image" type listed.

### Step 2: Configure the script

Open `attach_images_to_partdb.py` in any text editor and update the two
lines near the top:

```python
DEFAULT_PARTDB_URL = "http://YOUR-PART-DB:PORT"       # e.g. http://192.168.64.80:8085
DEFAULT_PARTDB_TOKEN = "tcp_YOUR_TOKEN_HERE"         # the token you just copied
```

> **Security**: this file now contains a secret. Treat it like a password.
> See the [Security](#security) section at the bottom for the full list
> of dos and don'ts.

---

## The full workflow (for your first import)

### Step A: Export your orders from AliExpress

1. Go to [aliexpress.com](https://www.aliexpress.com) → **My Orders**
2. Click **Export** (top-right of the orders table) → choose **CSV**
3. Save the file as `orders.csv` in the same folder as the scripts
   (e.g. `C:\Users\Amir\Downloads\aliexpress-import\orders.csv`)

### Step B: Run the conversion script

```cmd
cd C:\Users\Amir\Downloads\aliexpress-import
python aliexpress_to_partdb.py orders.csv
```

This produces, in the same folder:
- `partdb_import.csv` — the Part-DB-format CSV (one row per item)
- `partdb_images/` — folder of all downloaded product images
- `partdb_images/attachments.json` — index of which image goes with which part

You should see output like:
```
Loaded 553 order line(s).
All done.
  CSV written to:   C:\...\partdb_import.csv
  Images stored in: C:\...\partdb_images
  Images downloaded: 553/553 (0 failed)
```

### Step C: Import the CSV into Part-DB

1. Open Part-DB in your browser
2. In the left sidebar, click **Tools** → **Import parts** (or similar)
3. Click **Choose file** → select `partdb_import.csv`
4. **Important settings** in the import wizard:
   - ✅ **CSV delimiter**: `,` (comma — the script uses standard CSV)
   - ✅ **Create unknown datastructures**: **checked** (so the "New parts" category and any new suppliers are auto-created)
   - **Mode**: "Create new entries" (default) — see [FAQ](#faq) for re-import behavior
5. Click through the column mapping. The headers in `partdb_import.csv` are
   already named to match Part-DB's import schema, so the auto-map should
   fill in everything correctly
6. Click **Import** / **Start import**
7. Wait for the progress bar to finish

> ⚠️ **Watch out for re-imports** (see [FAQ](#faq) below). The default
> "Create" mode will make duplicates if you re-import the same CSV.

### Step D: Attach the images

```cmd
cd C:\Users\Amir\Downloads\aliexpress-import
python attach_images_to_partdb.py
```

You'll see one line per upload, e.g.:
```
[   1/553] ENS160 ENS160+AHT21 CARBON Dioxide CO2 e  OK  part 558: attached to part 558
[   2/553] TPS63020 Micro Power Module Automatic Vo  OK  part 559: attached to part 559
...
----------------------------------------------------------------------
Done.
  Uploaded:  553
  Skipped:    0  (already attached)
  No match:   0  (part not found in Part-DB)
  Failed:     0
```

For ~553 items at 0.15s between calls, expect roughly **1–2 minutes**.

The script is **idempotent** — running it again will skip parts that
already have an attachment with the same filename. So you can re-run it
anytime without creating duplicates.

### Step E: Verify

Open any of the imported parts in Part-DB — the product image should
appear in the **Attachments** tab (and as a thumbnail in the part list
if Part-DB auto-set it as the preview, which it usually does).

---

## For future use (when you buy more parts)

The next time you want to import a fresh AliExpress order:

1. Export the new `orders.csv` from AliExpress
2. If the new CSV contains only new parts (no overlap with the old import),
   you can just re-run **Steps B → D** above with the new file
3. If the new CSV might overlap with the old one, read the
   [Re-imports / dedup](#faq) section in the FAQ below

**Optional but recommended**: rotate your Part-DB API token periodically
to keep it from accumulating exposure. Just delete the old one in
Part-DB's API tokens page and paste the new one into the script.

---

## Script 1: `aliexpress_to_partdb.py`

### What it does
1. Reads `orders.csv` from AliExpress
2. Maps each row to a Part-DB part using the v2.15.x CSV import schema
3. Downloads every `Item image url` to `partdb_images/` (upgrades
   AliExpress's `_220x220.jpg` thumbnail URLs to full-size, falls back
   to the thumbnail on 404)
4. Writes `attachments.json` mapping each part to its image file(s)
5. Writes `partdb_import.csv` ready for the Part-DB import wizard

### Run command

```cmd
cd C:\Users\Amir\Downloads\aliexpress-import
python aliexpress_to_partdb.py orders.csv
```

### Options

```cmd
python aliexpress_to_partdb.py orders.csv -o my_parts.csv -d ./images
python aliexpress_to_partdb.py orders.csv --no-images       :: CSV only, skip downloads
```

### Customizing the default category

The default category for all imported parts is `New parts`. To change it
(for example to `Electronics/Sensors` or `Imported/2026-Q3`):

1. Open `aliexpress_to_partdb.py` in a text editor
2. Find the line near the top:
   ```python
   DEFAULT_CATEGORY = "New parts"
   ```
3. Change the string to whatever you want
4. Save the file

> The category is **auto-created** the first time you import, as long as
> "Create unknown datastructures" is checked in the Part-DB import wizard
> (see Step C above). Sub-categories use `->` as the separator
> (e.g. `Imported -> 2026 -> Q3`).

### Output files

| File | Contents |
|---|---|
| `partdb_import.csv` | The CSV you upload to Part-DB's import wizard |
| `partdb_images/<part-name>_<idx>.jpg` | One image per part, full size when available |
| `partdb_images/attachments.json` | Index: row → part name → image file path. Used by script 2 |

---

## Script 2: `attach_images_to_partdb.py`

### What it does
1. Reads `attachments.json` from script 1
2. For each entry, queries Part-DB for a part with the matching `name`
3. Auto-picks an attachment type whose name looks like "Image" /
   "Photo" / etc., or falls back to the first available type
4. Uploads the image as a PartAttachment
5. Skips parts that already have an attachment with the same filename
   (idempotent — safe to re-run)

### Run command (just run it)

```cmd
cd C:\Users\Amir\Downloads\aliexpress-import
python attach_images_to_partdb.py
```

The URL and token are read from the `DEFAULT_PARTDB_URL` and
`DEFAULT_PARTDB_TOKEN` constants at the top of the file. Override with
env vars (recommended for shared machines) or CLI flags (see below).

### Options

```cmd
:: Show all flags
python attach_images_to_partdb.py --help

:: Override the URL/token (e.g. for a different Part-DB instance)
python attach_images_to_partdb.py --url http://localhost:8000 --token tcp_xxx

:: Test with the first 3 entries, no changes made
python attach_images_to_partdb.py --limit 3 --dry-run

:: Run for real, only on the first 50
python attach_images_to_partdb.py --limit 50

:: List all attachment types in your Part-DB (useful for picking one)
python attach_images_to_partdb.py --list-attachment-types

:: Force a specific attachment type
python attach_images_to_partdb.py --attachment-type /api/attachment_types/1

:: Skip TLS verification (self-signed certs)
python attach_images_to_partdb.py --insecure
```

### Using env vars instead of editing the file (more secure)

```cmd
set PARTDB_URL=http://192.168.64.80:8085
set PARTDB_TOKEN=tcp_xxxxxxxx
python attach_images_to_partdb.py
```

In PowerShell:
```powershell
$env:PARTDB_URL = "http://192.168.64.80:8085"
$env:PARTDB_TOKEN = "tcp_xxxxxxxx"
python attach_images_to_partdb.py
```

---

## FAQ

### When I re-import the same CSV, does it skip existing parts?

**No, by default Part-DB will create duplicates.** The CSV importer's
default mode is "Create new entries" — it does not check for existing
parts.

For the AliExpress use case (one big import, plus small future imports
of new items), this is rarely a problem because the new orders contain
different parts. But if you ever need to re-import the same file (or a
file that overlaps with an old one), do one of these:

**Option 1 — Use the import wizard's matching mode**

In the Part-DB import wizard, look for a "Mode" or "Match existing
parts" option:
- **Create** (default) — always creates new parts → duplicates
- **Update** — updates matching parts, leaves the rest alone
- **Skip** — skips any part that already matches

If available, pick **Update** or **Skip** and choose the match key
(usually `name` works, but `manufacturer_product_number` is more
reliable for electronics).

**Option 2 — Delete duplicates manually after import**

After the import, in the Part-DB parts list, sort by name, find the
duplicates, select all → **Delete**. Tedious but works.

**Option 3 — Import to a temporary category, then move**

Import into a unique category (e.g. `Incoming/2026-08-15`), then
manually move parts to their final categories. Anything that was
already in the final categories will now appear twice, which is easy
to spot.

### The image script says "no attachment types defined"

You skipped [Step 1.5](#step-15-one-time-create-an-image-attachment-type)
or your Part-DB has no attachment types at all. Fix:

```cmd
python attach_images_to_partdb.py --list-attachment-types
```

If it prints nothing, go to **Admin → Attachment types** in Part-DB's
web UI and create at least one (recommended name: `Image`).

### The image script says "401 Unauthorized"

Token is wrong, expired, or your user lacks API permission.
- Open Part-DB → User menu → Settings → API tokens → create a new one
- Paste it into `DEFAULT_PARTDB_TOKEN` in the script (or set
  `PARTDB_TOKEN` env var)
- If the API tokens section is missing, ask your admin to grant
  "Miscellaneous → API access" + "Create API tokens" to your user

### The image script says "no match in Part-DB" for some rows

Either:
- You skipped Step C (CSV import) — the parts aren't in Part-DB yet
- The part name in Part-DB differs slightly from what's in the CSV
  (e.g. truncated, case-different, or with extra characters)
- You're looking at parts in a different Part-DB instance

The first one is most common. Run Step C, then re-run the image script —
it will skip already-uploaded ones and pick up where it left off.

### The connection fails / "cannot reach Part-DB"

- Make sure `http://YOUR-PART-DB:PORT` opens in your browser first
- If Part-DB is on HTTPS, change `DEFAULT_PARTDB_URL` to start with
  `https://` and possibly add `--insecure` if you have a self-signed cert
- If Part-DB is on a different machine, check the firewall / port
  forwarding

### Can I import other vendors (LCSC, Mouser, Digi-Key)?

The CSV format from other vendors is different. To support them you'd
need a new conversion script. The `aliexpress_to_partdb.py` script has
the AliExpress header names hard-coded near the top — you'd write a
parallel script with a different `AE = {...}` mapping. Happy to help if
you need this.

### How do I move all my AliExpress parts to a different category later?

In Part-DB, go to the part list, filter by `category = "New parts"`,
select all → **Edit** → change the category. Or use the Part-DB REST
API for bulk moves.

---

## Troubleshooting checklist

| Symptom | Likely cause | Fix |
|---|---|---|
| `File invalid. Please check that you have selected the right format!` in Part-DB import | CSV column names don't match Part-DB's schema | Re-run `aliexpress_to_partdb.py`; the current version uses the verified 2.15.x column names |
| `must be one of Orderdetail[] (string given)` | Old script version used `orderdetails` as a flat column | Use the latest `aliexpress_to_partdb.py` from this package |
| `KeyError: 'ordernumber'` at script startup | Old script version | Use the latest version from this package |
| `DeprecationWarning: 'maxsplit' is passed as positional argument` | Old script version (cosmetic) | Update to the latest version |
| Image script: `no attachment types` | Skipped Step 1.5 | See [this FAQ entry](#the-image-script-says-no-attachment-types-defined) |
| Image script: `no match in Part-DB` | Skipped CSV import step, or names differ | Run Step C first, or check Part-DB web UI |
| Image script: `401 Unauthorized` | Bad/expired token, or no API permission | Recreate token in Part-DB; check user has API permission |
| Image script: `SSL: CERTIFICATE_VERIFY_FAILED` | Self-signed HTTPS cert | Add `--insecure` flag |
| Some images fail to download (`404 Client Error`) | Seller removed the image from AliExpress CDN | Script falls back to the thumbnail; if even that fails, the part is created without an image — you can attach a manual photo later |

---

## Security

This package handles an API token that grants access to your Part-DB
under your user account. Please:

✅ **Do:**
- Keep the script folder on your local machine only
- Use a fresh token dedicated to this purpose
- Treat the file like a password — it should not leak
- Rotate (delete + recreate) the token periodically
- If you ever share your screen / codebase, blank the
  `DEFAULT_PARTDB_TOKEN` line first
- For a "cleaner" approach, use the `PARTDB_TOKEN` env var instead of
  baking it into the file

❌ **Don't:**
- Commit the folder to git (add a `.gitignore` containing `*.py` if you must)
- Sync the folder to cloud backups (OneDrive, Dropbox, Google Drive,
  iCloud) without first removing the token
- Share the folder publicly
- Re-use the same token for other applications
- Leave the file on a USB stick you might lose

### If your token leaks
1. In Part-DB: User menu → Settings → API tokens → delete the leaked one
2. Create a new token and paste it into the script
3. Check Part-DB's audit log for unexpected activity
4. Done — the new token works immediately

---

## Files at a glance

```
aliexpress-import\
├── README.md                          ← you are here
├── aliexpress_to_partdb.py            ← script 1: CSV + image downloader
├── attach_images_to_partdb.py         ← script 2: REST API uploader
├── orders.csv                         ← (your input) AliExpress export
├── partdb_import.csv                  ← (output) Part-DB-ready CSV
└── partdb_images\                     ← (output) downloaded images
    ├── ENS160_ENS160+AHT21_..._1.jpg
    ├── TPS63020_Micro_Power_..._2.jpg
    ├── …
    └── attachments.json               ← (output) image index for script 2
```

---

## Quick reference card

| Task | Command |
|---|---|
| Convert CSV + download images | `python aliexpress_to_partdb.py orders.csv` |
| Skip image download (CSV only) | `python aliexpress_to_partdb.py orders.csv --no-images` |
| Import to Part-DB | Use web UI: Tools → Import parts → upload `partdb_import.csv` |
| Attach all images | `python attach_images_to_partdb.py` |
| Test 3 images first | `python attach_images_to_partdb.py --limit 3 --dry-run` |
| List attachment types | `python attach_images_to_partdb.py --list-attachment-types` |
| Show all flags | `python attach_images_to_partdb.py --help` |

---

## Version notes

Tested with:
- AliExpress CSV export (as of mid-2026)
- Part-DB v2.15.0
- Python 3.10 / 3.11 on Windows 10/11

If Part-DB changes its CSV schema in a future version, you may need to
update the column names in `PARTDB_COLUMNS` near the top of
`aliexpress_to_partdb.py`. Check the official docs:
<https://docs.part-db.de/usage/import_export.html>