Composer archive only require-dev tree

how-to-archive-only-composer-dev-dependencies.md

How to archive only composer require-dev with all their dependencies

To optimize our CI builds we wanted to split our production dependencies from out dev dependencies. So that we would aquire two seperate archives which could be deployed seperatley.

For production we would only deploy production.tar.gz for testing we would deploy require.tar.gz and require-dev.tar.gz.

Unfortunately composer only comes with the composer install/update -no-dev that would ignore installation of dev dependencies.

Currently we are running two composer install while moving the vendor/ into vendor_prod/ and so forth using rsync to find the differenc between prod and dev installation and creating the proper tar balls. Sadly this creates a huge IO load.

Optimizing the IO load

To optimize the IO load I wanted to come up with a command that "only" installs dev dependencies, such as composer install --only-dev. As there is no such command I come up with a bash script oneliner which does this

cat composer.json | jq -r '."require-dev"' | sed --regexp-extended 's/\{|\}|\s+"([^"]+)":.*/\1/' | xargs -I {} composer show {} -t | sed --regexp-extended 's/[^a-z0-9]*([a-z0-9_-]+\/[a-z0-9_-]+).*|.*/vendor\/\1/' | sed '/^$/d' | sort | uniq | tar zcf require-dev.tar.gz --ignore-failed-read -T -

Required Tools

The only tool you would need is jq which can parse the json and extract the node of require-dev dependencies.

Breaking down the script

1. This will simply read your composer.json content to pipe it to jq

cat composer.json

2. Extract require-dev node from composer.json

jq -r '."require-dev"'

---
{
  "zendframework/zend-expressive-tooling": "^1.2",
  "behat/behat": "^3.5"
}

3. Use sed to extract only package names

sed --regexp-extended 's/\{|\}|\s+"([^"]+)":.*/\1/'

---
zendframework/zend-expressive-tooling
behat/behat

4. Use composer show X -t to get the tree of dependencies for each package

xargs -I {} composer show {} -t

---
zendframework/zend-expressive-tooling 1.2.1 Migration and development tooling for Expressive
├──ext-json *
├──ocramius/package-versions ^1.3
│  ├──composer-plugin-api ^1.0.0
│  └──php ^7.1.0
├──php ^7.1
├──symfony/console ^2.8 || ^3.0 || ^4.0
...

behat/behat 3.5.2 Description
├──symfony/console ^2.8 || ^3.0 || ^4.
...

5. Use sed again to extract every packagename with certain pattern and remove everything else (like extension or language dependencies). Also adding vendor/ to the resulting package name to reflect package path.

sed --regexp-extended 's/[^a-z0-9]*([a-z0-9_-]+\/[a-z0-9_-]+).*|.*/vendor\/\1/'

---
vendor/zendframework/zend-expressive-tooling

vendor/ocramius/package-versions


vendor/symfony/console

vendor/behat/behat
vendor/symfony/console

6. Remove empty lines using sed

sed '/^$/d'

---
vendor/zendframework/zend-expressive-tooling
vendor/ocramius/package-versions
vendor/symfony/console
vendor/behat/behat
vendor/symfony/console

7. Use sort and uniq to remove duplicate package names

sort | uniq

---
vendor/behat/behat
vendor/ocramius/package-versions
vendor/symfony/console
vendor/zendframework/zend-expressive-tooling

8. Pack every folder path into a output tar ball. This will also ignore every read package name that does not exist. For example if you are using roave/security-advisories as its only a meta package without actual files.

tar zcf require-dev.tar.gz --ignore-failed-read -T -