Archive for June, 2016

How to make AppDynamics work with OpsGenie


22 Jun

App Dynamics doesn’t directly support Ops Genie, and Ops Genie does not have a direct integration with App Dynamics. I found a post about adding a custom module to a dedicated App Dynamics controller that would work but most people don’t get a dedicated App Dynamics controller, it’s expensive and designed for large customers only.

So I was working on a shared multi-tenant SaaS app dynamics setup/account and I learned they support REST requests. OpsGenie also supports RESTful HTTP API calls. So we just need to make AppDynamics push HTTP REST requests to OpsGenie’s API. I couldn’t find any documentation/guides online on how to do this so I’m posting it here real quick.

Open up https://www.opsgenie.com/docs/web-api/alert-api#createAlertRequest (use this for reference)

  1. Login to OpsGenie
  2. Click on “Integrations” on the left and then click on “Add New Integration”
  3. Add an API integration and copy the API Key
  4. Name it something like “AppDynamics REST”
  5.  Click on Save Integration

Ok time to do the AppDynamics side

  1. Login to the AppDynamics
  2. Click on “Alert & Respond” at the top
  3. Click on “HTTP Request Templates”
  4. Click on +New
  5. Name could be something like “OpsGenie Create Alert”
  6. While not required, I recommend adding a Custom Templating Variable and calling it “apiKey” and then putting your API key you got from OpsGenie in there (it’s best to make the key a variable, easier to change in the future rather than hard coding it)
  7. Authentication should be NONE
  8. You can leave the “Custom Request Headers” section blank/alone
  9. Payload, change MIME Type to “application/json” and Payload Encoding can remain UTF-8
  10. Now comes the actual body of the message, as an example you can put in
{
"apiKey": "${apiKey}",
"source": "APPDYNAMICS",
"alias": "${latestEvent.id}",
"message": "${latestEvent.severity} - ${latestEvent.application.name} on ${latestEvent.node.name}",
"description": "${latestEvent.displayName} - AppDynamics has detected a ${latestEvent.severity} level alert with the ${latestEvent.application.name} application on the ${latestEvent.node.name} server. Please look into this.",
"note": "Please view the AppDynamics event that triggered this from: ${latestEvent.deepLink} for more details."
}

Just put that in the box.

11. In “Response Handling Criteria” for “Success Criteria” set status code to “200” and check “Expect Payload” and set Content Type to “application/json” (This tells AppDynamics that OpsGenie will respond with a 200 OK HTTP message and the format will be in json.
12. Click on “One Request Per Event”
13. Click on Save

 

So the main trick is the body of the message. Let’s go over it.

apiKey will be replaced with the apiKey variable you filled in at the top.

source can be anything, for my example I put in APPDYNAMICS.

alias is optional, if you don’t specify it OpsGenie will make up a random alias (which doubles as the Alert ID) but I figured it’s nice to make the OpsGenie Alert ID/Alias line up with the AppDynamics Event ID.

message Think of this like a subject. This is what will be in the actual alert, the alert that gets SMS/TXT messaged to the on-call’s phone, or the alert that gets read to the on-call over the phone. It’s limited to 140 characters.

description I treat this like the body of an email. This is where I can put in more details, you have a lot more room to work with. When the on-call person gets notified, they’ll see message but they’ll be given a link they can click on and that’ll bring them to the OpsGenie alert and that’ll show them the description.

note is equivalent to a comment. When it creates the alert in OpsGenie, message and description show up in the alert but then you can comment on the alert, so this will create the alert and immediately add a comment to it. The person checking the alert will also see this comment/note at the bottom in addition to what they see in the description.

You can play around with the variables and how things are formatted. Use that link I referenced (https://www.opsgenie.com/docs/web-api/alert-api#createAlertRequest) for examples of variables you can use.

HTTP Strict Transport Security (HSTS)


22 Jun

So a while back I was trying to get an A+ rating from ssllabs.com for glitchsys.com, because, why not?  I followed a guide on hardening NGINX for SSL and I went from a C to an A+. It was awesome.  However one of the things they had me add to my nginx.conf file was:

# Enable HTTP Strict Transport Security (HSTS)
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

Now I was vaguely aware of what this did, but not fully. Let me break this down for you. It will add that HTTP header to all HTTP requests.

  • So, 63072000 is in seconds, that large number translates to 730 days, or 2 years. So you’re telling the browser that for the next 2 years, always use https when accessing the domain. Now if somebody typed in “www.glitchsys.com” your browser may quickly make a HTTP connection, at least the first time, see that header and immediately reconnect via HTTPS and it will continue to do so for the next 2 years.
  • Also “includeSubdomains” means that *.glitchsys.com will always be accessed via HTTPS. So you could go to something like “blahblah.glitchsys.com” and your browser will immediately go to “https://blahblah.glitchsys.com” heck, you could even go to http://blahblah.glitchsys.com/ and it will immediately change to https://blahblah.glitchsys.com/ at least for the next 2 years. I use google apps so I had set “gmail.glitchsys.com” to be a CNAME to ghs.googlehosted.com which meant when you went to http://gmail.glitchsys.com/ you pretty much got redirected to gmail. However, thanks to that statement above now http://gmail.glitchsys.com/ resulted in the browser trying to go to https://gmail.glitchsys.com/ which went to the IP of google’s servers which don’t support https. Essentially, I completely mucked up gmail.glitchsys.com because that subdomain had to be http and not https. Oops.
    • I’m actually going to see if there is a way to “undo” HSTS for a specific sub domain. I’d like to have HSTS enabled for *.glitchsys.com but maybe I can arrange for specific sites like gmail.glitchsys.com to not use HSTS. I doubt I can do it though because…
  • And lastly, that “preload” action at the end turned out to be a lot more interesting than I thought. It turns out there is a chance somebody could theoretically intercept traffic to http://www.glitchsys.com/ and try to get rid of that Strict-Transport-Security (HSTS) line in the header, so to compensate you can add preload line which somehow gets your domain submitted to Google where it gets included in this large file and all future versions of Chrome will have this file hardcoded into it and will always go to your domain via HTTPS.  Let me repeat that, Chrome is now hardcoded to go to my site *.glitchsys.com via HTTPS and there’s nothing I can easily do to stop it. You actually have to remove permanent, set max-age to 0 seconds and submit a request to Google to have your domain removed and then in future versions of Chrome your domain won’t be included, which means it’d only affect people who download the new version of Chrome. A slow change over the course of many months.   So yeah, be careful with “permanent”.

Ok so Chrome does not store these settings, like if your domain must be HTTPS, in cache. You can clear your browsers Cache and it will still remember to go to your domain, like glitchsys.com, via HTTPS. Here’s a trick I learned:

Chrome, Opera: Cannot connect to the real <domain name>.

  1. In the address bar, type “chrome://net-internals/#hsts”.
  2. Type the domain name in the text field below “Delete domain”.
  3. Click the “Delete” button.
  4. Type the domain name in the text field below “Query domain”.
  5. Click the “Query” button.
  6. Your response should be “Not found”.

Very handy trick. However, for me, I tried to delete “glitchsys.com” and then I immediately did a query and it was still there! I tried over and over again, cleared my cache, tried again, no matter what I did I could not remove glitchsys.com; then I saw this in the query “static_upgrade_mode: STRICT" which I then learned was because I had done that "permanent" option above. 

Thanks to Arran Schlosberg and StackzOfZtuff over at StackExchange for making it clear. I’ll paste what was written:


If either of the static_upgrade_mode: or the dynamic_upgrade_mode: lines are set to STRICT then HSTS is enabled.

Dynamic

Dynamic means that the browser has been instructed to enable HSTS by an HTTP response header (served over TLS) similar to the following:

Strict-Transport-Security: max-age=157680000; includeSubDomains;

This is vulnerable to an attack whereby the very first time the browser requests the domain with http:// (not https://) an adversary intercepts the communication.

Static

In order to overcome this weakness we have the static mode which allows for hard-coding HSTS records directly into the browser’s source. The header is changed to indicate the administrator’s intention:

Strict-Transport-Security: max-age=157680000; includeSubDomains; preload

Note the inclusion of preload at the end. The domain is then submitted for manual review. If approved then it is added to the Chromium list which is also included in the Firefox, Safari, and IE 11+Edge lists.


So I went over to the Chromium list (https://chromium.googlesource.com/chromium/src/net/+/master/http/transport_security_state_static.json)  and it contained a long list of domains, a search revealed glitchsys.com was among them.  That’s why I could not get Chrome to “forget” glitchsys.com used HSTS no matter what I tried. Interesting. I guess I’m stuck using HTTPS whether I like it or not, hehe.

Deon's Playground

Placing whatever interests me and more