Memory leak in Appdynamics Ruby agent

Ruby Memory leak collections

### Ruby Agent Heapdump collection:

Preparing your rails application for heapdump Analysis.

1. In your rails application create initializer that initiates the ObjectSpace:
```
    #<Rails App>/config/initializers/heap_dump_tracing.rb
    require 'objspace'
    ObjectSpace.trace_object_allocations_start
```
2. Cleanup <project>/log directory for fresh trace level logs
    #config/appdynamics.yml
    app_name: xx
    tier_name: xx
    node_name: xxxx
    log_level: trace
    controller:
        host: xxxxx
        port: 443 # Optional
        account: xxxx
        access_key: xxxx
        use_ssl: true # Optional
        log_level: trace

3. Base Line without Appdynamics: Its good to have two virtual machine/Containers to sort out the base-line With Appdynamics Agent and Without Appdynamics. But if you have just 1 vm/Containers
you need to get those baseline with Appd and Without Appd one after the other. Also try to have one ruby process (Avoid Many workers getting involved) to have good outcome.
For below example I assume there is 1 MASTER and 1 PUMA Worker.

4. Follow below steps with Appdynamics Ruby agent and Without Appdynamics Ruby agent.

Lets divide our observation/Actions into
    1. Preload
    2. WithLoad
    3. PostLoad
    
    
-----------------------
### Preload : Even before load starts. collect and save heapdump at preload directory. Note: for every heapdump you collect give some justified time between heapdump collections.


create a Directory called "preload" - `mkdir preload`

bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("preload/ruby-heap-master-1.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("preload/ruby-heap-worker-1.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("preload/ruby-heap-master-2.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("preload/ruby-heap-worker-2.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("preload/ruby-heap-master-3.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("preload/ruby-heap-worker-3.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'

Screenshots:
    Screenshot of `top`, save it under preload directory
    Screenshot of `ps -ef | grep puma`, save it under preload directory
    screenshot of vm1/docker with appdynamics memory and cpu of controller.(Server Tab)
    requests per min screenshot
    
    
    
---------------------
### withload : With load being rushed to rails server. collect and save heapdump at with directory. Note: for every heapdump you collect give some justified time untill it reaches max observed memory utilization like (90% even before App server crashes).


create a Directory called "withload" - `mkdir withload`

bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("withload/ruby-heap-master-1.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("withload/ruby-heap-worker-1.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("withload/ruby-heap-master-2.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("withload/ruby-heap-worker-2.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("withload/ruby-heap-master-3.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("withload/ruby-heap-worker-3.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'

Screenshots:
    a. Screenshot of `top`, save it under withload directory
    b. Screenshot of `ps -ef | grep puma`, save it under withload directory
    c. vm1/docker with appdynamics memory and cpu screenshot of controller.(Server Tab)
    d. requests per min screenshot
---------------------

### Postload : Allow Application to cooldown. collect and save heapdump at with directory. Observe that to what extent memory consumption dropped post load is dropped.

create a Directory called "Postload" - `mkdir postload`

bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("postload/ruby-heap-master-1.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("postload/ruby-heap-worker-1.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("postload/ruby-heap-master-2.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("postload/ruby-heap-worker-2.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <MASTERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("postload/ruby-heap-master-3.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'
bundle exec rbtrace -p <WORKERPID> -e 'Thread.new{GC.start;require "objspace";io=File.open("postload/ruby-heap-worker-3.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'

Screenshots:
    a. Screenshot of `top`, save it under postload directory
    b. Screenshot of `ps -ef | grep puma`, save it under postload directory
    c. vm1/docker with appdynamics memory and cpu screenshot of controller.
    d. requests per min screenshot
---------------------

5. Submit all these to get reviewed by Ruby SME 
    We have three set of data to collect
    a. Without Appd Agent
        create a directory call "withoutAppd", Add below directories and zip it as withoutAppd.zip. Remember screenshots will also be present in respective directories.
            1. Preload
            2. WithLoad
            3. PostLoad

    b. With Appd Agent
        create a directory call "withAppd", Add below directories and zip it as withAppd.zip. Remember screenshots will also be present in respective directories.
            1. Preload
            2. WithLoad
            3. PostLoad

    c. Zip of clean trace level logs. zip of <project>/log