# PGC Parameter Troubleshooting
The commands listed in this document use BOSH CLI v2+ and CF CLI version 6.37.0+a40009753.2018-05-25+
## Background
In addition to standard PCF service parameters, Our recent releases have included the ability for customers to modify the PostgreSQL configuration parameters while creating or updating service instances.
In Crunchy PCF release version 03.090512.109, users could create a service instance with the `pgc_` parameters.
An example of creating a service instance with modifying the `max_wal_senders` and `log_rotation_size` PostgreSQL options is as follows:
```
$ cf create-service postgresql-9.5-odb small myService -c '
{
"db_name": "exampledb",
"db_username": "exampleuser",
"owner_name": "Example User",
"owner_email": "example.user@company.com"
"pgc_max_wal_senders": 20,
"pgc_log_rotation_size": 200000,
}
```
Starting in 04.x.001, users were able to pass those parameters to the `update-service` call to update the instance as they saw fit without having to recreate it.
Any PCF Service Parameter that is prefixed with `pgc_` refers to a parameter that would modify a PostgreSQL configuration option.
## Known Issue
**(As of 04 June 2018)**
Due to a bug in the [crunchy-postgresql-service-adapter](https://github.com/CrunchyData/crunchy-postgresql-service-adapter),
if a customer uses a `pgc_` parameter at any time, no further updates may be performed on the service instance.
This means that if a `pgc_` is used either during the creation of a service instance or used by calling the `cf update-service` command, that service instance will no longer accept any further `cf update-service` commands and will report a 502 error.
---
**NOTE:**
- The service instance continues to run after the `update-service` call fails.
- Other parameters can be used with no affect to the service instance.
- This issue is only related to parameters prefixed with `pgc_`.
---
As an example, if a customer required an update to a `pgc_` parameter for `max_wal_senders`,
using the service creation example from above, they would issue the command
```
$ cf update-service myService -c '{"pgc_max_wal_senders": 30}'
```
and the result code back to them would look like
```
Updating service instance myService as $USERNAME...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker error: Service cannot be updated at this time, please try again later or contact your operator for more information
```
If the customer looks at the service status, it will return a status message of `update failed`
```
$ cf service myService
Showing info of service myService in org pcfdev-org / space pcfdev-space as admin...
name: myService
service: postgresql-9.5-odb
tags:
plan: small
description: Crunchy PostgreSQL 9.5 On-Demand Managed Service
documentation:
dashboard:
There are no bound apps for this service.
Showing status of last operation from service myService...
status: update failed
message: Service broker error: Service cannot be updated at this time, please try again later or contact your operator for more information
started: 2018-05-31T17:04:04Z
updated: 2018-05-31T17:04:04Z
```
**NOTE:** This will happen for any parameters updates including non-`pgc_` parameters and regardless of the inclusion in the original setup. A service cannot be fixed by re-running `cf update-service` with a non-pgc parameter.
## Resolution
The service instance, even if showing `update failed`, will continue to run.
- If the parameter is not required, the customer should be advised that that no further action is required.
- If the customer must have the updated PostgreSQL configuration parameter(s) applied to the service instance, there are three possible resolutions to the situation.
- After any of these resolutions are applied, the `pgc_` parameters should not be used until the next Crunchy PCF update is released and installed.
**NOTE:** Pivotal advises against performing the following manual intervention resolutions, unless absolutely necessary. The recommended advice to a customer is to create a new service instance with the desired parameters, transfer the data, and rebind any applications.
### Automatically Fix All Instances
The first possible remedy is to have the customer's platform administration team run the `upgrade-all-service-instances` errand against the broker. This will remove the `pgc_` references and reset the instances to defaults.
The functionality of the [Pivotal On Demand Broker](https://github.com/pivotal-cf/on-demand-service-broker) is such that when the `upgrade-all-service-instances` errand is called, all of the manifests will have the `pgc_` parameters removed and set back to platform defaults.
To run the errand, the platform administration team (not the end-user client) needs to execute:
`bosh -e $ENV -d crunchy-postgresql-on-demand-broker run-errand upgrade-all-service-instances`
**NOTE:** This will iterate over all service instances and will cause interruption to any existing service instances.
This methodology will also be used to apply the PCF Team's official [Code Fix](#code-fix) when it is released.
### Manually Redeploying An Instance With Updated Parameters
These steps will allow a customer to update the service instance with the desired parameters. However, it will still be in a state in which the `update-service` call will fail and any alterations must be made by these steps and with the assistance of the customer's platform administration team.
1. The customer must obtain the GUID associated with the service instance using the Cloud Foundry Command Line.
- `cf service $SERVICE_NAME --guid`
- This will be referenced as `$GUID` for the following steps.
1. **Platform Administrator Required:** Download the manifest in a temporary YAML file.
- `bosh -e $ENV -d $service-instance_$GUID manifest > /tmp/service-instance_$GUID.yml`
- Example: `bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 manifest > /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml`
1. Using an editor that saves in Unix format, open the manifest file, modify the parameters needed, and save.
```yml
- name: postgresql
properties:
postgresql:
config:
```
- An example for changing the example provided in the background section would be to change the `max_wal_senders` property under the config from 20 to 30.
```yml
- name: postgresql
properties:
postgresql:
config:
- max_wal_senders: 20
+ max_wal_senders: 30
```
1. After the manifest has been updated, the Platform Administration team can update the instance using the deploy command
- `$ bosh -e vbox -d $service-instance_$GUID /tmp/service-instance_$GUID.yml`
#### Here is an example update that modifies the `max_wal_senders` and adds in a value for `wal_keep_segments`.
```
$ bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 deploy /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml
Using environment '192.168.50.6' as client 'admin'
Using deployment 'service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825'
instance_groups:
- name: postgresql
properties:
postgresql:
config:
- max_wal_senders: "<redacted>"
+ max_wal_senders: "<redacted>"
+ wal_keep_segments: "<redacted>"
Continue? [yN]: y
Task 20
Task 20 | 21:25:39 | Preparing deployment: Preparing deployment (00:00:01)
Task 20 | 21:25:42 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 20 | 21:25:42 | Updating instance postgresql: postgresql/e9f2e1ba-886e-4238-af1f-bc87a7769efc (00:01:30))
Task 20 | 21:27:12 | Updating instance postgresql: postgresql/7b3e1007-fd6d-4287-9695-0c5fa4b27d35 (1) (00:03:27)
Task 20 Started Thu May 31 21:25:39 UTC 2018
Task 20 Finished Thu May 31 21:30:39 UTC 2018
Task 20 Duration 00:05:00
Task 20 done
Succeeded
```
### Manually Resetting An Instance Back To Defaults
These steps will allow a customer to reset the instance back to the default settings and all the use of the `update-service` call one time.
1. The customer must obtain the GUID associated with the service instance using the Cloud Foundry Command Line.
- `cf service $SERVICE_NAME --guid`
- This will be referenced as `$GUID` for the following steps.
1. **Platform Administrator Required:** Download the manifest in a temporary YAML file.
- `bosh -e $ENV -d $service-instance_$GUID manifest > /tmp/service-instance_$GUID.yml`
- Example: `bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 manifest > /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml`
1. Using an editor that saves in Unix format, open the manifest file, and remove any `pgc_` parameters that been added to the instance.
```yml
- name: postgresql
properties:
postgresql:
config:
```
1. After the manifest has been updated, the Platform Administration team can update the instance using the deploy command
- `$ bosh -e vbox -d $service-instance_$GUID /tmp/service-instance_$GUID.yml`
#### Here is an example update that removes the `max_wal_senders`.
```
$ bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 deploy /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml
Using environment '192.168.50.6' as client 'admin'
Using deployment 'service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825'
instance_groups:
- name: postgresql
properties:
postgresql:
config:
- max_wal_senders: "<redacted>"
Continue? [yN]: y
Task 107
Task 107 | 21:25:39 | Preparing deployment: Preparing deployment (00:00:01)
Task 107 | 21:25:42 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 107 | 21:25:42 | Updating instance postgresql: postgresql/e9f2e1ba-886e-4238-af1f-bc87a7769efc (00:01:30))
Task 107 | 21:27:12 | Updating instance postgresql: postgresql/7b3e1007-fd6d-4287-9695-0c5fa4b27d35 (1) (00:03:27)
```
## Code Fix
In our next release, `04.090513.002`, the service adapter will accommodate the
adjusted paths and will, after running the `upgrade-all-service-instances`
process in PCF to update all of the services instances to the new release, the
`update-service` call will work properly.