Skip to content

Instantly share code, notes, and snippets.

@dlopes7

dlopes7/lab_draft.md

Last active June 19, 2018 17:34
Show Gist options
  • Save dlopes7/d54ca5b879b0571d6fc2fb96bd682d45 to your computer and use it in GitHub Desktop.
Save dlopes7/d54ca5b879b0571d6fc2fb96bd682d45 to your computer and use it in GitHub Desktop.
Download ZIP
EUM Practical Lab Draft
Raw
lab_draft.md

Step 1 - Setting up the environment

Ravello Environment

  1. Create a new application from the Blue Print PS - BRUM Lab
  2. Configure both servers to use a Private Key that you can access
    • If you do not have a Private Key, create a Key Par on Ravello.
  3. Publish and Start the application and wait for the servers to come up

Architecture

The environment is compose of 2 Servers:

"App"

"Controller and EUM"

  • AppD 4.4.3 Controller
  • AppD 4.4.3 EUM Server
  • AppD 4.4.3 Events Service

Credentials

  1. Controller
    • user: admin
    • password: appd123
    • root password: appd123
    • OS user: centos
  2. App
    • OS user: centos

Starting the AppDynamics environment

The Controller should start automatically

  1. SSH to the controller server using your private key
  2. Start the Events Service sudo service appdes start
  3. Wait for the ES to start service appdes status

[centos@controller ~]$ service appdes status AppDynamics - Events Service: Running
AppDynamics - Events Service ping/: Success

  1. Start the EUM Server sudo service appdeum start
  2. Wait for the EUM Server to start sudo service appdeum status

[centos@controller ~]$ sudo service appdeum status .
AppDynamics - EUM Server: Running .
AppDynamics - EUM Server eumcollector/: Up .
AppDynamics - EUM Server eumaggregator/: Up .

Starting the demo application

Shopizer is a Spring-Boot ecommerce Java application. We can start it using Maven, and it will run with an embbeded Tomcat server. There is a convenience "start.sh" script included, but you can also start the app manually if you'd like.

  1. Login to the App Server using your private key
  2. Move into the lab folder cd /opt/lab
  3. Start the application with a ./start_app.sh
  4. Tail appd.log file until you see Started ShopApplication in XX seconds (JVM running for XX) in the file

tail -f app.log

With these steps done, you should have:

  • The controller available at http://<controller_domain>:8090/
  • The EUM server listening at http://<controller_domain>:7001/
  • The Java App Tomcat listening at http://<app_domain>:8080/
  • The Apache Server listening at http://<app_domain>/ and redirecting traffic to the Tomcat

Generate & Apply the temp license

We need to generate temp license and apply to the controller & EUM.

Genearte the temp license

  1. Go to https://portal.appdynamics.com/licenses/testlicense/#/
  2. Select 'Enable EUM' checkbox
  3. Make sure 'SAAS EUM Collector' is slected.
  4. Change from 'lite' to 'pro' for Browser Real User Monitoring.
  5. Enter 1 in '# of Units'
  6. Click on 'Provision'. This should generate a temp license file.
  7. Click on 'Download License File'.

Apply the temp license to the controller

  1. scp license.lic file to the controller in /opt/appdynamics/components/controller.

Apply the temp license to the EUM

  1. ssh to the EUM server.
  2. cd /opt/appdynamics/components/EUM/eum-processor/bin
  3. ./provision-license /opt/appdynamics/components/controller/license.lic - Make sure there are no errors in provisioning the license.

Verify the license

  1. login to the controller UI.
  2. Go to Settings -> License
  3. Verify license under User Experience

Update Beacon Location

We need to Configure your controller Beacon and EUM Cloud settings to point to your EUM server since the EUM Server url will be unique for each lab instance.

  1. Go to: http://<controller_host>:8090/controller/admin.jsp
  2. Login with the password appd123
  3. Under Controller Settings, change the variables:
    • eum.beacon.host = http://<controller_eum_server_host>:7001
    • eum.beacon.https.host = https://<controller_eum_server_host>:7002
    • eum.cloud.host = http://<controller_eum_server_host>:7001

Step 2 - Enabling EUM Monitoring

Before we can monitor the application, we need to create the EUM Application on the Controller.

  1. Create an EUM Application for Shopizer
    • Navigate to "User Experience", and Click on "Add App"
    • Select "Create an Application manually"
    • Call it "Shopizer - BRUM"

Lab 1 - Apache Injection

Unfortunately, our app is not compatible with automatic injection.

One of the options we have in this scenario is using Apache to do the javascript injection for us, our app server has Apache 2.4 installed, so we are good to go.

In this case, we will instruct the Apache Web Server to intercept the response html pages, and inject our javascript snippet at the page using the mod_filter and mod_substitute modules. This technique is also possible with any web server or load balancer that supports body rewriting, like Nginx, HAProxy, IIS, BigIP F5, etc.

  1. We need to instruct apache to inject our javascript snippet into our app

    • Lets edit our apache config file for our app /etc/httpd/conf.d/ecommerce.conf
    • The snippet is already there and commented, but we need to update it with our EUM Server hostname and App Key
      1. Copy the App Key from the User Experience Application you created earlier
      2. Replace config.appKey = 'OLD_APP_KEY' on the ecommmerce.conf file, with your correct App Key from step 1
        • This can be done by running sudo sed -i -e 's/OLD_APP_KEY/<YOU_APP_KEY_HERE>/g' /etc/httpd/conf.d/ecommerce.conf
        • You can also edit the file with sudo vim /etc/httpd/conf.d/ecommerce.conf and replace the entry manually
      3. Replace OLD_EUM_DNS on the ecommmerce.conf file, with your EUM (Controller) DNS from Ravello.
        • This can be done by running sudo sed -i -e 's/OLD_EUM_DNS/<YOUR_RAVELLO_CONTROLLER_DNS>/g' /etc/httpd/conf.d/ecommerce.conf
        • You can also edit the file with sudo vim /etc/httpd/conf.d/ecommerce.conf and replace the entry manually
      4. Now uncomment lines 11, 12, 13, 14 and 15 from the /etc/httpd/conf.d/ecommerce.conf file and save the changes
    • restart apache with sudo service httpd restart

  2. The apache webserver listens on port 80, let's test the injection

    • Navigate to http://<your_app_vm_dns>/ on your browser
    • Inspect the page source, verify that the apache server successfully injected your javascript snippet
      • Before before
      • After after
    • Generate some load by navigating to different pages for some minutes
    • Verify that the data is being sent to the EUM server by looking at the Developers Tools of your browser, or by navigating to the EUM Application on the controller.

Lab 2 - Manual Injection

If for some reason you are not able to use automatic injection, and there is no Web Server or Load Balancer available to inject, we can use Manual Injection, in which we generate a Javascript snippet ask the developer to include it in the pages we wish to monitor.

  1. Inside the EUM Application screen you created earlier, let's generate the javascript snippet

    • Navigate to "Configuration" and then "Configure and download the Javascript agent"
    • On the step 1 make sure "AppDynamics hosts all Javascript Agent files from cdn.appdynamics.com" is selected
    • Click on "Save Config & Generate HTML Snippet"
    • Copy the resulting html snippet

  2. Deploying our snippet

    • Manual injection will require a code change, so we need to stop our app, make changes to the code, rebuild and redeploy

      • ssh to the "App" server
      • Navigate to /opt/lab and execute ./stop_app.sh
      • Let's edit our main template page to include the javascript snippet:
      • vim /opt/lab/shopizer/sm-shop/src/main/webapp/pages/shop/templates/generic/catalogLayout.jsp
      • Paste the AppDynamics agent snippet inside the <head> tag
      • Save the file.
      • snippet

    • Now we need to rebuild and redeploy the app

      • cd /opt/lab
      • ./rebuild_and_redeploy.sh
      • Wait for the build to finish and for the app to start back up again

    • Now let's verify that our code change worked

      • Open your browser to 'http://<app_server_domain>:8080' (Note: do not navigate to port 80 because Apache will inject the javascript from the first exercise, if you want to use port 80, comment the injection from the apache config and restart apache)
      • Check the webpage source code to see the snippet
      • Go to developer mode (F12 or Ctrl + Shift + I)
      • Open the network tab, refresh the page, and make sure POSTS to /adrum appear
        • adrum

  3. Generate load by navigating around the pages for a couple of minutes

    • Check that you can see load on the Controller EUM App you created earlier

Lab 3 - Capturing User Data

The customer wants to know which products are being added to the shopping cart.
We can grab that information using our Javascript agent and send it to AppDynamics as BiQ data.

For this lab we will need to update the javascript snippet, there are three ways to accomplish this:

  • Just editing the /opt/lab/shopizer/sm-shop/src/main/webapp/pages/shop/templates/generic/catalogLayout.jsp file, springboot will apply the changes automatically, no need to redeploy.
  • Editing the apache configuration file and restarting apache
  • Using Josh Sho's chrome extension

Before proceeding:

  • If you plan to use the apache or the Chrome extension route, comment the lines you added to /opt/lab/shopizer/sm-shop/src/main/webapp/pages/shop/templates/generic/catalogLayout.jsp.

  1. Understanding the application behaviour and where to capture data

    • Navigate to your application page at http://<your_app_vm_dns>:8080/
    • Open your browser Developer Tools, and switch to the Network tab
    • Add a product to your shopping cart.
      • Observe that an AJAX Request is made to /shop/cart/addShoppingCartItem
      • If you inspect the request, you will see that it sends the data we need on the querystring, 'productId' and 'quantity'
      • In the next steps we will create the logic to capture those two parameters.

  2. Code to capture the parameters

    function getParameterByName(name, url) {
    if (!url) url = window.location.href;
    
    name = name.replace(/[\[\]]/g, "\\$&");
    var regex = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)"),
        results = regex.exec(url);
        
    if (!results) return null;
    
    if (!results[2]) return '';
    return decodeURIComponent(results[2].replace(/\+/g, " "));
}
    • And then, let's add our new configuration object to the window['adrum-config'] object
    window['adrum-config'] = {
    userEventInfo: {
        "Ajax": function (context) {

            // If we are here, we are doing a POST to the cart
            if (context.url.indexOf('addShoppingCartItem') > -1 && context.method === 'POST'){

                // The value we want is on the URL parameters
                var productId = getParameterByName('productId', context.url);
                var quantity = getParameterByName('quantity', context.url);
                return {
                    userDataLong: {
                        productId: Number(productId),
                        quantity: Number(quantity)
                    }
                }
            }

            // If we get here, we are not doing an AJAX that is interesting for us
            return {};
        }
    }
};
    • All we need to do is paste both functions before our window['adrum-start-time'] = new Date().getTime(); line and save the file.
    • If you are using Apache, you will need to compress the entire code to one line before pasting it into the Substitute directive. I recommend doing manual injection here.
    • Now add a product to the card, and check that the data is sent to AppDynamics on the POST to /adrum, you should see an element udl that represents the User Data Long we added.
    • post

Note: In AppDynamics version 4.4.3 we cannot easily capture data from the POST PAYLOAD and send it as custom data. The Shopizer code was changed for this lab, to send the data on the querystring instead.

Lab 4 - Custom Geo Location

Sometimes you will need to setup custom Geo Locations for a customer. This could happen because of many reasons, maybe they have a wide range of users coming from internal IP addresses, or maybe some internet IP addresses are not mapped to locations. In this lab we will practice on how to create custom geo locations by editing the geo-ip-mappings.xml file that comes bundled with the EUM Server.

  1. Check what Geo Location your EUM Server is assigning to you by visiting http://<eum-controller-server-dns>:7001/eumcollector/whoami
    • This will also tell you what your current Internet IP Address is. Save that information for now.
  2. SSH into your EUM Server
  3. Stop the eum server sudo service appdeum stop
  4. Edit the file /opt/appdynamics/components/EUM/eum-processor/bin/geo-ip-mappings.xml. This can be done with vim.
    • Study the format of the file, there should be some sample entries commented out
    • Create a new entry for your current IP Address, the Country and Region have to be present here if you want it on the EUM Map, you can use any city name.
    • geo
  5. Restart the eum server sudo service appdeum stop sudo service appdeum start
  6. Wait a couple of minutes for the server to start, and then visit http://<eum-controller-server-dns>:7001/eumcollector/whoami again, make sure you see your new location
    • geodone

Lab 5 - Troubleshoot Correlation Issues

AppDynamics provides correlation between End User Experience and APM Apps. Technically, the two are correlated through a response cookie called ADRUM that is injected into the response. Sometimes for security reasons, only certain cookies are allowed on an environment, and our cookie could be blocked. In the next lab we will follow steps to troubleshoot and fix correlation issues.

  1. First we need to enable correlation
    • Navigate to Applications, Shopizer
    • Configuratuion, User Experience App Integration
    • Make sure that Enable Business Transaction Correlation is checked
    • Leave all settings as they are
  2. Navigate to http://<app_server_dns>:8080/
  3. Open your browser Developer Tools, on the homepage request, you should see an ADRUM_BT* cookie on the response
    • This means that AppD is sending the correlation header
    • correlation
  4. Navigate to http://<app_server_dns>/
    • Here the correlation cookies are not present.
    • This means that something is removing them from the response.
  5. The only difference between 3. and 4. is the Apache Server
    • Let's investigate the apache, edit the file sudo vim /etc/httpd/conf/httpd.conf
    • There is a line in this file that is blocking our Cookie Header edit Set-Cookie ADRUM.*?=[^;]*?$ ""
    • Comment this line with #, save the file and restart apache sudo service httpd restart
  6. Verify that correlation is being applied again
  7. Generate traffic and check your Browser Page snapshots, some should have a corresponding APM Snapshot
    • correlation_snapshots

Usually it would not be Apache blocking this cookie, but a firewall or another security appliance. The idea is to work with the customer to figure out if Cookies are being stripped somewhere between the application server and the user browser.

Lab 6 - Name Pages based on POST Parameter

The login transaction has a parameter which tells us the Store Code that the user is logging in to. The customer wants to track the Ajax Response times for the different store codes by using different Page Names for them.

Note that this is not a common usecase for Page Names, usually we would capture this data with analytics, for the purpose of this lab we are using this data to learn how we would go about using something from a POST Payload as the name of the page.

  1. The first step is to learn how the data can be obtained, navigate to http://app_server_dns:8080/shop/customer/customLogon.html
    • Open the development tools of your browser, and perform a login
    • Observe the parameters and what their names are for the POST that is done to http://app_server_dns:8080/shop/customer/logon.html
    • post_parameter
  2. From the first step, we realize we want to use the parameter storeCode when there is a POST to /shop/customer/logon
  3. AppDynamics provides an API to intercept those calls and capture context detail that can then be used to name pages.
    • Documentation can be found here
  4. Let's edit our snippet to include the captured POST payload, this requires some javascript knowledge and can be done in several ways, here is a sample implementation of a function that grabs a parameter from a JSON payload:
function getParameter(paramKey, data) {

    if (typeof data === 'string') {
        var fields = data.split("&");
        for (var i = 0; i < fields.length; i++) {
            var keyAndValue = fields[i].split('=');
            if (keyAndValue.length > 1) {
                var key = keyAndValue[0],
                    value = keyAndValue[1];
                if (key === paramKey)
                    return value;
            }
        }
    }
}
  1. Bring it together on your snippet, like so:
window['adrum-config'] = {
    xhr: {
        parameter: {
            urls: [{
                pattern: '.*'
            }],
            getFromBody: function (data) {
                var storeCode = getParameter('storeCode', data);
                if (storeCode) {
                    return {
                        storeCode
                    }
                }
            }
        }
    }
};
  1. Use the full snippet, and perform a login again, check that the value is being sent to AppDynamics on the POST to /adrum:
    post_with_data
  2. On the controller interface, let's configure EUM to use our parameter as Page Name
    • End User Experience > Your App > Configuration > Instrumentation
    • Create a new Ajax Include Rule
    • Rule Name: logon
    • Criteria: Ends With logon.html
    • Check POST
    • POST Parameters to use in Page Name (Optional): storeCode
    • sample_config
  3. Generate more load by performing several logins, doesn't matter if the username/password is incorrect
    • Note that it might take a couple of minutes for the new Ajax Detection Rule to propagate
  4. Check that the new Page Name was created in AppDynamics with the storeCode by navigating to "Pages & AJAX Requests"
    • The StoreCode we use is "Default" so you should see a new page name with the name "default"
    • OPTIONAL: You can register new stores on the admin page and validate the different names
      • The url for that is http://<you_app_host>:8080/admin
      • username: admin, password: password
      • Navigate to Store, Dropdown Menu, Create a store
    • result_eum
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment