NXLog User Guide

130. Microsoft Azure and Office 365

This add-on is available for purchase. For more information, please contact us.

This NXLog add-on can retrieve information about various user, admin, system, and policy actions and events from Microsoft Azure and Office 365. Once configured, the add-on prints Syslog events, each with a JSON payload, to standard output for processing by NXLog. The log sources are selectable from the following reports corresponding to the supported Microsoft REST-based APIs:

Azure Active Directory Graph Reports and Events API: accountProvisioningEvents, allUsersWithAnomalousSignInActivityEvents, applicationUsageDetailEvents, applicationUsageSummary, applicationUsageSummaryEvents, auditEvents, b2cUserJourneyEvents, b2cUserJourneySummaryEvents, cloudAppDiscoveryEvents, compromisedCredentials, groupActivitySummary, irregularSignInActivityEvents, mimSsgmGroupActivityEvents, mimSsprActivityEvents, mimSsprRegistrationActivityEvents, signInsAfterMultipleFailuresEvents, signInsFromIPAddressesWithSuspiciousActivityEvents, signInsFromMultipleGeographiesEvents, signInsFromPossiblyInfectedDevicesEvents, signInsFromUnknownSourcesEvents, ssgmGroupActivityEvents, ssprActivityEvents, ssprRegistrationActivityEvents, threatenedCredentials, userActivitySummary
Office 365 Management Activity API: Audit.AzureActiveDirectory, Audit.Exchange, Audit.SharePoint
Office 365 Service Communications API: CurrentStatus, HistoricalStatus, Messages, Services

For more information about the log sources, see the relevent link(s) below:

130.1. Installation

Install the add-on. For example, with dpkg:
```
# dpkg -i nxlog-msazure-<version>.deb
```
If the previous command exits non-zero, resolve any missing dependencies:
```
# apt-get -f install
```

The installation can be found under /opt/nxlog-msazure.

Note	The `nxlog-msazure` add-on depends on `nxlog` or `nxlog-trial`.

130.2. Prerequisites

In order to use the tool you will need the following prerequisites, which are covered in the following sections:

An Azure Active Directory application with appropriate permissions
client_id
client_secret (key)
tenant_domain
tenant_id
certthumbprint
certkeyfile (certkey.pem)

Most of the configuration will require accessing the Azure Management Portal with a tenant user that has the necessary permissions to make the configuration changes.

130.2.1. Create an Azure AD Application to Access the APIs

To access information from your directory, you must create an application in your Azure Active Directory tenant and grant it appropriate read permissions to access the data.

To create an application and grant permission, carry out the steps described in Prerequisites to access the Azure Active Directory reporting API. Record the domain name (tenant_domain), the client_id, and the client_secret (key).
1. Select the Active Directory instance.
2. Open the Applications tab and add a new application by clicking Add at the bottom.
3. Choose to Add an application my organization is developing, name the application, select the Web application and/or Web API type, and set the application properties.
4. Open the Configure tab for the new application and copy the Client ID.
5. Generate a client key with a duration of 2 years, click Save, and copy the key (client_secret).
Find the tenant_id by clicking on the View endpoints on the bottom of the Azure classic portal page. This will display multiple endpoint URLs, each containing a UUID value. Record the UUID in the OAUTH 2.0 TOKEN ENDPOINT URL as tenant_id.

130.2.2. Grant Permissions to the Application

Under permissions to other applications, click Add application.
Select Office 365 Management APIs and Windows Azure Active Directory.
Click on the Application permissions popdown menu and select all read permissions.
Save the changes by clicking the Save button.

130.2.3. Generate and Set Up an X.509 Certificate

The log collection process uses service-to-service calls via the Microsoft REST-based APIs, so it is important to generate and set up an X.509 certificate for authenticating to the service. A gencertkey.sh script is provided for Linux that can be used to simplify the process. It creates the private key-pair in a certkey.pem file in the working directory.

The gencertkey.sh script depends on openssl and uuidgen. Install the corresponding packages if necessary. For example, on Debian-based platforms:
```
# apt install openssl uuid-runtime
```

Generate the certificate with the gencertkey.sh script.

$ ./gencertkey.sh
Generating a RSA private key
............+++++
................................................+++++
writing new private key to 'certkey.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
ThumbPrint:0nFt3fB0JP7zuSmHaRQtmsFNYqo=
"keyCredentials": [
{
    "customKeyIdentifier":"0nFt3fB0JP7zuSmHaRQtmsFNYqo=",
    "keyId":"629ab88d-1059-454b-b258-4ca05b46dee4",
    "type":"AsymmetricX509Cert",
    "usage":"Verify",
    "value":"MIIDXTCCAkWgAwIBAgIJAP+XrnwhAxjOMA0GCSqGSIb3DQEBCwUAMEUxCzAJB..."
}
],

Record or save the following output which will be needed later:
- the output key file certkey.pem (certkeyfile),
- the base64-encoded certificate fingerprint value after ThumbPrint: (certthumbprint), and
- the KeyCredentials portion (which will be used in the steps immediately following).
Select Manage Manifest in the bar at the bottom and click Download Manifest.

Edit the downloaded manifest file and replace the empty KeyCredentials with the previously generated output.

From

"keyCredentials": [],

"keyCredentials": [
{
    "customKeyIdentifier":"0nFt3fB0JP7zuSmHaRQtmsFNYqo=",
    "keyId":"629ab88d-1059-454b-b258-4ca05b46dee4",
    "type":"AsymmetricX509Cert",
    "usage":"Verify",
    "value":"MIIDXTCCAkWgAwIBAgIJAP+XrnwhAxjOMA0GCSqGSIb3DQEBCwUAMEUxCzAJB..."
}
],

Save the modified manifest and upload it.

130.3. Parameters

If the script is started without any parameters, it will return the following help text:

$ ./msazure-pull.pl
NXLOG | Azure AD, Office 365 Management and Office 365 Service Communications API Reports Parser Utility 1.0
Copyright (c) 2016 NXLog Ltd., All rights reserved.
Usage: ./msazure-pull.pl
       ./msazure-pull.pl --verbose=true|false|yes|no
       ./msazure-pull.pl --infinity=true|false|yes|no
       ./msazure-pull.pl --skip_state_file=true|false|yes|no
       ./msazure-pull.pl --sleep=nnnnn
       ./msazure-pull.pl --working_directory=/from/this/script/runs/
       ./msazure-pull.pl --add_syslog_header=true|false|yes|no
       ./msazure-pull.pl --client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
       ./msazure-pull.pl --client_secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
       ./msazure-pull.pl --tenant_domain=contoso.onmicrosoft.com
       ./msazure-pull.pl --tenant_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
       ./msazure-pull.pl --certthumbprint=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
       ./msazure-pull.pl --certkeyfile=certkey.pem
       ./msazure-pull.pl --start_date=YYYY-MM-DDTHH:MM:SSZ|amonthago|aweekago|yesterday
       ./msazure-pull.pl --end_date=YYYY-MM-DDTHH:MM:SSZ|amonthago|aweekago|yesterday|now
       ./msazure-pull.pl --top=nnn
       ./msazure-pull.pl --management_activity_sources=Audit.Exchange,Audit.SharePoint,Audit.AzureActiveDirectory
       ./msazure-pull.pl --service_communication_operations=Services,CurrentStatus,HistoricalStatus,Messages
       ./msazure-pull.pl --azure_ad_reports=applicationUsageSummary,applicationUsageDetailEvents,applicationUsageSummaryEvents,b2cUserJourneySummaryEvents,b2cUserJourneyEvents,cloudAppDiscoveryEvents,userActivitySummary,groupActivitySummary,mimSsgmGroupActivityEvents,ssgmGroupActivityEvents,mimSsprActivityEvents,ssprActivityEvents,mimSsprRegistrationActivityEvents,ssprRegistrationActivityEvents,threatenedCredentials,compromisedCredentials,auditEvents,accountProvisioningEvents,signInsFromUnknownSourcesEvents,signInsFromIPAddressesWithSuspiciousActivityEvents,signInsFromMultipleGeographiesEvents,signInsFromPossiblyInfectedDevicesEvents,irregularSignInActivityEvents,allUsersWithAnomalousSignInActivityEvents,signInsAfterMultipleFailuresEvents|list
       ./msazure-pull.pl --applicationId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

The burn-in configuration file, msazure-pull.bic, contains the parameters that generally do not need to be adjusted. The mandatory parameters and source parameters (queries) can be specified on the command line.

130.3.1. Mandatory Parameters

The script requires the following parameters. These values were recorded in the Prerequisites section.

--client_id=
--client_secret=
--tenant_id=
--certthumbprint=
--tenant_domain=
--certkeyfile=certkey.pem
--working_directory=

The full command should be similar to this:

$ ./msazure-pull.pl --client_id=912497ba-9780-46bc-a6a6-3a56a4c14278 \
      --client_secret=X4wlwAQ44+JHk4i/cxdpZX37XbHz9DQwXsmt/LL07UM=   \
      --tenant_id=e681b493-14a8-438b-8bbf-d65abdc826c2               \
      --certthumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=                  \
      --tenant_domain=contoso.onmicrosoft.com                        \
      --certkeyfile=certkey.pem                                      \
      --working_directory=/opt/nxlog/msazure-pull

However, additional parameters need to be provided to specify data sources. For example, here is the above command with three debug parameters added (for more information about these, see the Optional Parameters section):

$ ./msazure-pull.pl [...] \
      --verbose=true      \
      --infinity=false    \
      --skip_state_file=true
Loading burn-in configuration file: /opt/nxlog/msazure-pull/msazure-pull.bic
Processing command line parameters.
Error. There is no enabled source or misconfiguration.
Terminated. Good Bye.

This is normal because the data sources have not been specified.

130.3.2. Source Parameters

To specify the data sources, use the following parameters:

Azure Active Directory Graph Reports and Events API: --azure_ad_reports= (and --applicationId, required only for the applicationUsageDetailEvents report)
Office 365 Management Activity API: --management_activity_sources=
Office 365 Service Communications API: --service_communication_operations=

The available data types that the tool can pull are listed in the overview, and are also listed by the script if it is executed without any parameter (or in case of a misconfiguration).

130.3.2.1. Configure the Azure Active Directory API

With --azure_ad_reports=list, the tool will print a list of all available reports from the Azure Active Directory API (the following is an excerpt):

$ ./msazure-pull.pl --client_id=912497ba-9780-46bc-a6a6-3a56a4c14278 \
      --client_secret=X4wlwAQ44+JHk4i/cxdpZX37XbHz9DQwXsmt/LL07UM=   \
      --tenant_id=e681b493-14a8-438b-8bbf-d65abdc826c2               \
      --certthumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=                  \
      --tenant_domain=contoso.onmicrosoft.com                        \
      --certkeyfile=certkey.pem                                      \
      --working_directory=/opt/nxlog/msazure-pull                    \
      --verbose=true                                                 \
      --infinity=false                                               \
      --azure_ad_reports=list
Loading burn-in configuration file: /opt/nxlog/msazure-pull/msazure-pull.bic
Processing command line parameters.
Got OAuth2 access token from https://login.windows.net/.
Running query (1/1):
Without any parameter.
Request: https://graph.windows.net/contoso.onmicrosoft.com/reports/?api-version=beta
Processing responses:  without any parameter.
Received 25 rows.
There are 25 new rows among them.
<13>2016-09-01T12:00:31Z contoso.onmicrosoft.com {"LicenseRequired":"False","Name":"applicationUsageDetailEvents","service_operation":""}
<13>2016-09-01T12:00:31Z contoso.onmicrosoft.com {"LicenseRequired":"True","Name":"applicationUsageSummaryEvents","service_operation":""}
<13>2016-09-01T12:00:31Z contoso.onmicrosoft.com {"LicenseRequired":"False","Name":"b2cUserJourneySummaryEvents","service_operation":""}
[...]
Number of expired rows: 0, number of removed rows: 0.
State of Graph API | Azure Active Directory ( 25 lines ) saved to: /opt/nxlog/msazure-pull/azure.dat

If you run it again, the response will change to this:

$ ./msazure-pull.pl [...]
Loading burn-in configuration file: /opt/nxlog/msazure-pull/msazure-pull.bic
Processing command line parameters.
State of Graph API | Azure Active Directory ( 25 lines ) loaded from: /opt/nxlog/msazure-pull/azure.dat
Got OAuth2 access token from https://login.windows.net/.
Running query (1/1):
Without any parameter.
Request: https://graph.windows.net/contoso.onmicrosoft.com/reports/?api-version=beta
Processing responses:  without any parameter.
Received 25 rows.
There is no new row among them.
Number of expired rows: 0, number of removed rows: 0.
State of Graph API | Azure Active Directory ( 25 lines ) saved to: /opt/nxlog/msazure-pull/azure.dat

This is because the program saves the state of queries to state files to avoid printing the same data again. The state files (azure.dat, o365manact.dat, and o365sercom.dat) are stored in the working_directory.

Now use the --azure_ad_reports= parameter to specify three queries:

$ ./msazure-pull.py [...] \
      --azure_ad_reports=signInsAfterMultipleFailuresEvents,\
signInsFromMultipleGeographiesEvents,\
irregularSignInActivityEvents
[...]
Running query (1/3): signInsFromMultipleGeographiesEvents
[...]
Running query (2/3): irregularSignInActivityEvents
[...]
Running query (3/3): signInsAfterMultipleFailuresEvents
[...]

The full response should be something like this:

$ ./msazure-pull.py [...] \
      --azure_ad_reports=signInsAfterMultipleFailuresEvents,\
signInsFromMultipleGeographiesEvents,\
irregularSignInActivityEvents
Loading burn-in configuration file: /opt/nxlog/msazure-pull/msazure-pull.bic
Processing command line parameters.
State of Graph API | Azure Active Directory ( 10 lines ) loaded from: /opt/nxlog/msazure-pull/azure.dat
Got OAuth2 access token from https://login.windows.net/.
Running query (1/3): signInsFromMultipleGeographiesEvents
With the following parameter: '&$top=50'.
Request: https://graph.windows.net/contoso.onmicrosoft.com/reports/signInsFromMultipleGeographiesEvents?api-version=beta&$top=50
Processing responses: signInsFromMultipleGeographiesEvents with the following parameter: '&$top=50'.
Received 5 rows.
There are 5 new rows among them.
<13>2016-08-09T08:17:49Z contoso.onmicrosoft.com {"displayName":"John Doe","estimatedTravelHours":2,"firstSignInFrom":"Szolnok, Szolnok, HU","id":"2016-08-09T08:17:49.0000000e681b493-14a8-438b-8bbf-d65abdc826c21003BFFD97C2458416
3.172.152.231","secondSignInFrom":"Paris, Paris, FR","service_operation":"signInsFromMultipleGeographiesEvents","timeBetweenSignIns":"00:00:27","timeOfSecondSignIn":"2016-08-09T08:17:49Z","userName":"johndoe@contoso.onmicrosoft.com"}
[...]
Running query (2/3): irregularSignInActivityEvents
With the following parameter: '&$filter=eventTime%20ge%20<amonthago>%20and%20eventTime%20le%20<now>'.
Request: https://graph.windows.net/contoso.onmicrosoft.com/reports/irregularSignInActivityEvents?api-version=beta&$filter=eventTime%20ge%202016-08-01T12:40:33Z%20and%20eventTime%20le%202016-09-01T12:40:33Z
Processing responses: irregularSignInActivityEvents with the following parameter: '&$filter=eventTime%20ge%20<amonthago>%20and%20eventTime%20le%20<now>'.
Received 0 rows.
Running query (3/3): signInsAfterMultipleFailuresEvents
With the following parameter: '&$filter=firstSuccessfulSignIn%20ge%20<amonthago>%20and%20firstSuccessfulSignIn%20le%20<now>'.
Request: https://graph.windows.net/contoso.onmicrosoft.com/reports/signInsAfterMultipleFailuresEvents?api-version=beta&$filter=firstSuccessfulSignIn%20ge%202016-08-01T12:40:34Z%20and%20firstSuccessfulSignIn%20le%202016-09-01T12:40:34Z
Processing responses: signInsAfterMultipleFailuresEvents with the following parameter: '&$filter=firstSuccessfulSignIn%20ge%20<amonthago>%20and%20firstSuccessfulSignIn%20le%20<now>'.
Received 1 row.
There is a new row among them.
<13>2016-08-09T08:16:59Z contoso.onmicrosoft.com {"displayName":"John Doe","failedAttemptsToSignIn":23,"firstSuccessfulSignIn":"2016-08-09T08:16:59Z","id":"2016-08-09T08:16:59.0000000e681b493-14a8-438b-8bbf-d65abdc826c21003BFFD97C24584","service_operation":"signInsAfterMultipleFailuresEvents","userName":"johndoe@contoso.onmicrosoft.com"}
Number of expired rows: 10, number of removed rows: 10.
State of Graph API | Azure Active Directory ( 6 lines ) saved to: /opt/nxlog/msazure-pull/azure.dat

For each query, values will be printed for the total number of received rows and the number of those rows which are new. Syslog events (each containing a JSON payload) will be printed for each new row.

Received 5 rows.
There are 5 new rows among them.

During testing, remove the state files from the working_directory with rm o365sercom.dat o365manact.dat azure.dat or simply use --skip_state_file=true, which will be useful anytime you need a special report or a simple list. In these situations you can also use --infinity=false to turn off the script’s endless loop.

The --applicationId parameter is a special one that must be specified in order to query the Azure Active Directory applicationUsageDetailEvents report.

130.3.2.2. Configure the Office 365 Management Activity API

This works similar to the Azure Active Directory API, but in debug mode the messages are different.

$ ./msazure-pull.pl [...]                                      \
      --verbose=true                                           \
      --infinity=false                                         \
      --management_activity_sources=Audit.AzureActiveDirectory
Loading burn-in configuration file: /opt/nxlog/msazure-pull/msazure-pull.bic
Processing command line parameters.
Loading Office 365 certkeyfile.
Getting Office 365 access token.
Got Office 365 access token.
Started subscription for Office 365 Audit.AzureActiveDirectory.
Office 365 Management Activity API
Listing content from 'https://manage.office.com/api/v1.0/e681b493-14a8-438b-8bbf-d65abdc826c2/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2016-08-31T13:16:31Z&endTime=2016-09-01T13:16:31Z'.
Got 51 contents for Audit.AzureActiveDirectory.
Fetching content (1/50) from 'https://manage.office.com/api/v1.0/e681b493-14a8-438b-8bbf-d65abdc826c2/activity/feed/Audit/221342003578$221342003578$e681b49314a8438b8bbfd65abdc826c2aad2016083113$3978d419adfc4224a8d183e1176036a1$04'.
<13>2016-08-31T12:21:09 manage.office.com {"Actor":[{"ID":"johndoe@contoso.onmicrosoft.com","Type":5},{"ID":"1003BFFD97C24584","Type":3},
[...]
Got 50 total events for Audit.AzureActiveDirectory.
Total number of active unique events for all sources: 50
Number of new (printed) events from all sources: 50
Number of events in the local buffer: 50
State of Office 365 Management Activity API ( 50 lines ) saved to: /opt/nxlog/msazure-pull/o365manact.dat

If the script is executed again, it will not print the same Syslog events again, because they have already been printed.

130.3.2.3. Configure the Office 365 Service Communications API

$ ./msazure-pull.pl [...]                         \
      --verbose=true                              \
      --infinity=false                            \
      --service_communication_operations=Messages
Loading burn-in configuration file: /opt/nxlog/msazure-pull/msazure-pull.bic
Processing command line parameters.
Loading Office 365 certkeyfile.
Getting Office 365 access token.
Got Office 365 access token.
Office 365 Service Communications API | Messages
Fetching content from 'https://manage.office.com/api/v1.0/e681b493-14a8-438b-8bbf-d65abdc826c2/ServiceComms/Messages'.
Got 28 event(s).
<13>2016-07-27T19:00:00Z manage.office.com {"ActionType":null,"AffectedTenantCount":6631344,"AffectedUserCount":null,"AffectedWorkloadDisplayNames":[],"AffectedWorkloadNames":[],"Classification":"Advisory","EndTime":"2016-08-08T21:00:00Z","Feature":"Access","FeatureDisplayName":"E-Mail and calendar access","Id":"EX73176","ImpactDescription":null,"IsDismissed":false,"IsRead":false,"LastUpdatedTime":"2016-08-13T18:10:18.403Z","MessageType":"Incident","Messages":[{"MessageText":"Current Status: We're deploying the fix to the affected environment and the deployment is approximately 68 percent complete.
[...]
Total number of active unique events for all sources: 28
Number of new (printed) events from all sources: 28
Number of events in the local buffer: 28
State of Office 365 Service Communications API ( 28 lines ) saved to: /opt/nxlog/msazure-pull/o365sercom.dat

130.3.3. Optional Parameters

These parameters are not mandatory, and in most cases it is not necessary to set them. They have already been set in the burn-in configuration.

Three parameters control the range of events queried: top, start_date, and end_date.

--start_date=YYYY-MM-DDTHH:MM:SSZ|amonthago|aweekago|yesterday
--end_date=YYYY-MM-DDTHH:MM:SSZ|amonthago|aweekago|yesterday|now: The start_date and end_date parameters specify the time range of content to return. These parameters work with all Office 365 reports and most of the Azure Active Directory reports. Where start/end ranges are not supported, the script will use top. The amonthago, aweekago, yesterday, and now values are dynamic and will be calculated in every loop.
--top=n: The top parameter works only with Azure Active Directory reports and events. It returns a subset of the entries for the given report, consisting of the first n entries, where n is a positive integer. top=5 returns the 5 most recent audit report events. top will be overridden where start_date and end_date can be used—top is lower priority.

Example 627. Pulling Reports From the Last 24 Hours

To pull reports from the last 24 hours, use --start_date=yesterday --end_date=now.

There are several ways to find out which reports use top and which use start/end ranges:

The script’s debug output messages:

Running query (1/1): signInsAfterMultipleFailuresEvents
With the following parameter: '&$filter=firstSuccessfulSignIn%20ge%20<yesterday>%20and%20firstSuccessfulSignIn%20le%20<now>'.
Request: https://graph.windows.net/contoso.onmicrosoft.com/reports/signInsAfterMultipleFailuresEvents?api-version=beta&$filter=firstSuccessfulSignIn%20ge%202016-08-31T14:15:07Z%20and%20firstSuccessfulSignIn%20le%202016-09-01T14:15:07Z

The burn-in configuration file:

{
    "name": "applicationUsageSummaryEvents",
    "api-version": "beta",
    "datetime_field_name": "eventTime",
    "parameters": [{
    "filter": "eventTime ge <startdate> and eventTime le <enddate>"
}

Microsoft’s documentation

Remaining optional parameters include the following:

--add_syslog_header=true|false|yes|no: Enable or disable the Syslog header.
--infinity=true|false|yes|no: Indicates that the script should never stop and should pull logs in an endless loop. The default is true. This can be set to false for special cases or debugging, when the script should run once and then exit.
--skip_state_file=true|false|yes|no: If true, the script will neither read from nor write to the state files. The default is false.
--sleep=n: The script will sleep n seconds between loops.
--verbose=true|false|yes|no: For debugging; if true, provides as much detail as possible about what the script is doing. The default is false. In normal mode, the script should print only events, logs, and reports (data it retrieves from the APIs). The script emits all diagnostic messages to standard error.

130.4. Examples

Example 628. Collecting All Available Reports

With these parameters, msazure-pull.pl collects all available reports (except for applicationUsageDetailEvents) from all sources. In each loop, events are collected from the last 24 hours. For reports that do not support time ranges, the 50 most recent events are collected.

$ ./msazure-pull.pl --client_id=912497ba-9780-46bc-a6a6-3a56a4c14278 \
      --client_secret=X4wlwAQ44+JHk4i/cxdpZX37XbHz9DQwXsmt/LL07UM=   \
      --tenant_id=e681b493-14a8-438b-8bbf-d65abdc826c2               \
      --certthumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=                  \
      --tenant_domain=contoso.onmicrosoft.com                        \
      --certkeyfile=certkey.pem                                      \
      --working_directory=/opt/nxlog/msazure-pull                    \
      --management_activity_sources=Audit.Exchange,Audit.SharePoint,Audit.AzureActiveDirectory \
      --service_communication_operations=Services,CurrentStatus,HistoricalStatus,Messages \
      --azure_ad_reports=applicationUsageSummary,applicationUsageSummaryEvents,b2cUserJourneySummaryEvents,b2cUserJourneyEvents,cloudAppDiscoveryEvents,userActivitySummary,groupActivitySummary,mimSsgmGroupActivityEvents,ssgmGroupActivityEvents,mimSsprActivityEvents,ssprActivityEvents,mimSsprRegistrationActivityEvents,ssprRegistrationActivityEvents,threatenedCredentials,compromisedCredentials,auditEvents,accountProvisioningEvents,signInsFromUnknownSourcesEvents,signInsFromIPAddressesWithSuspiciousActivityEvents,signInsFromMultipleGeographiesEvents,signInsFromPossiblyInfectedDevicesEvents,irregularSignInActivityEvents,allUsersWithAnomalousSignInActivityEvents,signInsAfterMultipleFailuresEvents

Example 629. Collecting applicationUsageDetailEvents

To query the applicationUsageDetailEvents report, the --applicationId parameter must be specified. The applicationId can be discovered by querying the applicationUsageSummary report and looking for appId in the JSON payload.

$ ./msazure-pull.pl --client_id=912497ba-9780-46bc-a6a6-3a56a4c14278 \
      --client_secret=X4wlwAQ44+JHk4i/cxdpZX37XbHz9DQwXsmt/LL07UM=   \
      --tenant_id=e681b493-14a8-438b-8bbf-d65abdc826c2               \
      --certthumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=                  \
      --tenant_domain=contoso.onmicrosoft.com                        \
      --certkeyfile=certkey.pem                                      \
      --working_directory=/opt/nxlog/msazure-pull                    \
      --infinity=no                                                  \
      --add_syslog_header=no                                         \
      --skip_state_file=true                                         \
      --azure_ad_reports=applicationUsageSummary
[...]
{"appId":"00000002-0000-0ff1-ce00-000000000000","service_operation":"applicationUsageSummary","usageData":[{"dateTime":"2016-08-23T00:00:00Z", [...]
{"appId":"4b233688-031c-404b-9a80-a4f3f2351f90","service_operation":"applicationUsageSummary","usageData":[{"dateTime":"2016-08-23T00:00:00Z", [...]
{"appId":"a7c97cbe-3d06-4c22-bfd5-9a2619d32daf","service_operation":"applicationUsageSummary","usageData":[{"dateTime":"2016-08-09T00:00:00Z", [...]
[...]

Now the appId values can be used with the --applicationId parameter to query the applicationUsageDetailEvents report.

$ ./msazure-pull.pl [...]
      --infinity=no                                   \
      --add_syslog_header=no                          \
      --skip_state_file=true                          \
      --azure_ad_reports=applicationUsageDetailEvents \
      --applicationId=00000002-0000-0ff1-ce00-000000000000,4b233688-031c-404b-9a80-a4f3f2351f90,a7c97cbe-3d06-4c22-bfd5-9a2619d32daf
[...]
{"applicationId":"00000002-0000-0ff1-ce00-000000000000","eventTime":"2016-08-23T15:50:00Z","id":"e681b493-14a8-438b-8bbf-d65abdc826c2_00000002-0000-0ff1-ce00-000000000000_1003BFFD97C24584","maxActivityTimeStamp":"2016-08-23T15:50:00Z","numberOfClicks":15,"service_operation":"applicationUsageDetailEvents","totalDay30Count":15,"totalDay7Count":null,"totalHour24Count":null,"userName":"John Doe"}
{"applicationId":"4b233688-031c-404b-9a80-a4f3f2351f90","eventTime":"2016-08-23T10:08:05Z","id":"e681b493-14a8-438b-8bbf-d65abdc826c2_4b233688-031c-404b-9a80-a4f3f2351f90_1003BFFD97C24584","maxActivityTimeStamp":"2016-08-23T10:08:05Z","numberOfClicks":11,"service_operation":"applicationUsageDetailEvents","totalDay30Count":11,"totalDay7Count":null,"totalHour24Count":null,"userName":"John Doe"}
{"applicationId":"a7c97cbe-3d06-4c22-bfd5-9a2619d32daf","eventTime":"2016-08-09T08:19:49Z","id":"e681b493-14a8-438b-8bbf-d65abdc826c2_a7c97cbe-3d06-4c22-bfd5-9a2619d32daf_1003BFFD97C24584","maxActivityTimeStamp":"2016-08-09T08:19:49Z","numberOfClicks":4,"service_operation":"applicationUsageDetailEvents","totalDay30Count":4,"totalDay7Count":null,"totalHour24Count":null,"userName":"John Doe"}
[...]

130.5. NXLog Configuration

Once its has been verified that all the required logs are collected properly, the --verbose, --infinity, and --skip_state_file switches can be removed and NXLog can be configured. The certkey.pem file and the burn-in configuration msazure-pull.bic file need to be placed in the --working_directory.

Example 630. Testing With the CurrentStatus Report

The Office 365 Service Communications API CurrentStatus report may be helpful for testing; it will continually provide the current status of the services.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  <Input msazurepull>
    Module  im_exec
    Command /opt/nxlog-msazure/bin/msazure-pull.sh
    Arg     --client_id=912497ba-9780-46bc-a6a6-3a56a4c14278
    Arg     --client_secret=X4wlwAQ44+JHk4i/cxdpZX37XbHz9DQwXsmt/LL07UM=
    Arg     --tenant_id=e681b493-14a8-438b-8bbf-d65abdc826c2
    Arg     --certthumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=
    Arg     --tenant_domain=contoso.onmicrosoft.com
    Arg     --certkeyfile=certkey.pem
    Arg     --working_directory=/opt/nxlog/msazure-pull
    # Continually report the current services status
    Arg     --service_communication_operations=CurrentStatus
    Exec    parse_syslog();
</Input>

<Output out>
    Module  om_file
    File    "/tmp/out.log"
</Output>

Example 631. Collecting Azure Active Directory and Office 365 Events

This configuration queries most of the available reports from the supported Azure and Office 365 REST-based APIs—only the Azure applicationUsageDetailEvents and Service Communications CurrentStatus reports are excluded.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

  <Input msazurepull>
    Module  im_exec
    Command /opt/nxlog-msazure/bin/msazure-pull.sh
    Arg     --client_id=912497ba-9780-46bc-a6a6-3a56a4c14278
    Arg     --client_secret=X4wlwAQ44+JHk4i/cxdpZX37XbHz9DQwXsmt/LL07UM=
    Arg     --tenant_id=e681b493-14a8-438b-8bbf-d65abdc826c2
    Arg     --certthumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=
    Arg     --tenant_domain=contoso.onmicrosoft.com
    Arg     --certkeyfile=certkey.pem
    Arg     --working_directory=/opt/nxlog/msazure-pull
    Arg     --azure_ad_reports=applicationUsageSummary,\
applicationUsageSummaryEvents,b2cUserJourneySummaryEvents,b2cUserJourneyEvents,\
cloudAppDiscoveryEvents,userActivitySummary,groupActivitySummary,\
mimSsgmGroupActivityEvents,ssgmGroupActivityEvents,mimSsprActivityEvents,\
ssprActivityEvents,mimSsprRegistrationActivityEvents,\
ssprRegistrationActivityEvents,threatenedCredentials,compromisedCredentials,\
auditEvents,accountProvisioningEvents,signInsFromUnknownSourcesEvents,\
signInsFromIPAddressesWithSuspiciousActivityEvents,\
signInsFromMultipleGeographiesEvents,signInsFromPossiblyInfectedDevicesEvents,\
irregularSignInActivityEvents,allUsersWithAnomalousSignInActivityEvents,\
signInsAfterMultipleFailuresEvents
    Arg     --management_activity_sources=Audit.Exchange,Audit.SharePoint,\
Audit.AzureActiveDirectory
    Arg     --service_communication_operations=Services,HistoricalStatus,Messages
    Exec    parse_syslog();
</Input>

<Output out>
    Module  om_file
    File    "/tmp/out.log"
</Output>